Optimizing access to federation infrastructure-based resources

ABSTRACT

The present invention extends to methods, systems, and computer program products for optimizing access to federation infrastructure-based resources. Various different layers within a federation infrastructure can signal location change events indicating the hosting location and/or access location for a resource is to be optimized. In response to a location change event, redirection information for accessing the resource is updated within the federation infrastructure. The redirection information is used to redirect resource access requests to appropriately optimized locations within the federation infrastructure. Redirecting resource access requests reduces communication within the federation infrastructure.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. patent applicationSer. No. 10/971,451, filed Oct. 22, 2004, and entitled “RendezvousingResource Requests With Corresponding Resources”, which is hereinincorporated by reference in its entirety. This application is acontinuation-in-part of U.S. patent application Ser. No. 11/936,556,filed Nov. 7, 2007, and entitled “Data Consistency Within A FederationInfrastructure”, which is herein incorporated by reference in itsentirety. This application is a continuation-in-part of U.S. patentapplication Ser. No. 12/556,399, filed Sep. 9, 2009, and entitled“Organizing Resources Into Collections To Facilitate More Efficient AndReliable Access”, which is herein incorporated by reference in itsentirety.

BACKGROUND Background and Relevant Art

Computer systems and related technology affect many aspects of society.Indeed, the computer system's ability to process information hastransformed the way we live and work. Computer systems now commonlyperform a host of tasks (e.g., word processing, scheduling, accounting,etc.) that prior to the advent of the computer system were performedmanually. More recently, computer systems have been coupled to oneanother and to other electronic devices to form both wired and wirelesscomputer networks over which the computer systems and other electronicdevices can transfer electronic data. Accordingly, the performance ofmany computing tasks are distributed across a number of differentcomputer systems and/or a number of different computing environments.

Some network environments include a federation infrastructure (e.g., aring) of federated nodes. The federation infrastructure, or at least acomponent of, can provide a hosting environment for applications and/orservices that have clients outside of the hosting environment (or“external clients”). In these and other similar hosting environments,stateful resources (e.g., a ring node id ownership token) may be sharedacross a plurality of external clients.

However, external clients often have no indication of the configurationof nodes within the hosting environment. For example, external clientsare typically unaware of the actual node hosting a resource they mayneed. Thus, when external clients initiate communication with the hostenvironment, for example, to request a resource, they typically do sothrough a random node (sometimes referred to as an “entrée node”). Theentrée node then routes stateful resource related messages within thehost environment (possibly over multiple hops) towards the node hostingthe requested resource. Among other things, routing of stateful resourcerelated messages within the hosting environment results in inefficientuse of hosting environment resources.

BRIEF SUMMARY

The present invention extends to methods, systems, and computer programproducts for optimizing access to federation infrastructure-basedresources. Embodiments of the invention including optimizing access to aring infrastructure resource. In some embodiments, a user request toaccess a ring infrastructure resource is received. The user request issent from a requesting user component.

It is detected that the user request is not directed to an optimizedlocation for accessing the ring infrastructure resource within the ringinfrastructure based on characteristics of the requesting usercomponent. An indication of an appropriately optimized location for therequesting user component to access the ring infrastructure resource issent based on the characteristics of the requesting user component.

In other embodiments, a ring infrastructure detects a component request.The component request is for optimizing access to one or more ringinfrastructure resources associated with a component on a ring of nodes.The optimized access is to reduce inter-node communication between nodeson the ring of nodes.

It is determined that at least one of the hosting location of and anaccess location for an associated ring infrastructure resource, selectedfrom among the one or more ring infrastructure resources, is notoptimized on the ring of nodes. The determination that the hostinglocation is not optimized is based on one or more users of the componenthaving a common interest in the associated resource. It is indicatingthat at least one of the hosting location of and the access location forthe associated ring infrastructure resource on the ring of nodes is tobe optimized to reduce inter-node communication costs between nodes onthe ring nodes in response to the component request.

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

Additional features and advantages of the invention will be set forth inthe description which follows, and in part will be obvious from thedescription, or may be learned by the practice of the invention. Thefeatures and advantages of the invention may be realized and obtained bymeans of the instruments and combinations particularly pointed out inthe appended claims. These and other features of the present inventionwill become more fully apparent from the following description andappended claims, or may be learned by the practice of the invention asset forth hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features of the invention can be obtained, a moreparticular description of the invention briefly described above will berendered by reference to specific embodiments thereof which areillustrated in the appended drawings. Understanding that these drawingsdepict only typical embodiments of the invention and are not thereforeto be considered to be limiting of its scope, the invention will bedescribed and explained with additional specificity and detail throughthe use of the accompanying drawings in which:

FIGS. 1A and 1B illustrate an example federation infrastructure thatfacilitates providing optimized access to federationinfrastructure-based resources.

FIG. 2 illustrates an example of different layers within a federationinfrastructure.

FIG. 3 illustrates an example federation infrastructure that facilitatesoptimizing a hosting location for a federation infrastructure-basedresource.

FIG. 4 illustrates a flow chart of an example method for providingoptimized access to federation infrastructure-based resources.

FIG. 5 illustrates a flow chart of an example method for configuringoptimized access to federation infrastructure-based resources.

FIG. 6 illustrates an example of a federation infrastructure.

FIG. 7 illustrates an example of a computer architecture thatfacilitates routing request indirectly to partners.

FIG. 8 illustrates an example binary relationship between nodes in afederation infrastructure in the form of a sorted list and correspondingring.

FIG. 9 illustrates an example ring of rings that facilitates proximalrouting.

FIG. 10 illustrates an example proximity induced partition tree of ringsthat facilitates proximal routing.

FIG. 11A illustrates an example of a node establishing membership withinan existing federation.

FIG. 11B illustrates an example of nodes in a federation infrastructureexchanging messages.

FIG. 12 illustrates an example of a message model and related processingmodel.

FIG. 13 illustrates an example of a number of liveness interactions thatcan occur between a function layer and an application layer.

FIG. 14 illustrates an example of messages forming part of arequest-response message exchange pattern are routed across nodes on aring.

FIG. 15A illustrates an example ring architecture that facilitates onenode monitoring another (e.g., subject) node.

FIG. 15B illustrates an example ring architecture that facilitates twonodes monitoring each other.

FIG. 15C illustrates an example ring architecture that facilitatesarbitration when mutually monitoring nodes can each report that theother node is suspected of failing.

FIG. 16A illustrates an example ring architecture that facilitatesrouting a message in accordance with a cached two-way agreement.

FIG. 16B illustrates an example ring architecture that facilitatesrouting a message in accordance with multiple cached two-way agreements.

FIGS. 17A through 17D illustrate an example ring architecture thatfacilitates formulating a cached two-way agreement.

FIG. 18 illustrates an example ring architecture that facilitatesjoining of a node to a ring of nodes within a federation with ringconsistency.

FIG. 19 illustrates an example state diagram for a joining node joininga ring of nodes with ring consistency.

FIG. 20 illustrates an example state diagram for a leaving node leavinga ring of nodes with ring consistency.

FIG. 21 illustrates an example ring architecture that facilitatesmaintaining replica set and data consistency within a federation.

FIG. 22 illustrates an exemplary sequence of operations.

FIG. 23 illustrates an exemplary system for accessing data from areplica set within a federation infrastructure.

FIG. 24 illustrates an example of a namespace federation infrastructure.

FIG. 25 illustrates an example of a computer architecture thatfacilitates routing request indirectly to partners.

FIG. 26 illustrates an example of a binary relation between namespacemanagers in a namespace federation infrastructure.

FIG. 27 illustrates an example of an architecture that facilitatesintegrating a namespace federation infrastructure with other protocols.

FIG. 28 illustrates an example of a namespace federation infrastructurefrom provider and collection views of a namespace.

FIG. 29 illustrates an example namespace federation infrastructure witha resource made available in a plurality of namespaces.

FIG. 30 illustrates an example schema taxonomy that can be used todescribe a resource.

DETAILED DESCRIPTION

The present invention extends to methods, systems, and computer programproducts for optimizing access to federation infrastructure-basedresources. Embodiments of the invention including optimizing access to aring infrastructure resource. In some embodiments, a user request toaccess a ring infrastructure resource is received. The user request issent from a requesting user component.

It is detected that the user request is not directed to an optimizedlocation for accessing the ring infrastructure resource within the ringinfrastructure based on characteristics of the requesting usercomponent. An indication of an appropriately optimized location for therequesting user component to access the ring infrastructure resource issent based on the characteristics of the requesting user component.

In other embodiments, a ring infrastructure detects a component request.The component request is for optimizing access to one or more ringinfrastructure resources associated with a component on a ring of nodes.The optimized access is to reduce inter-node communication between nodeson the ring of nodes.

It is determined that at least one of the hosting location of and anaccess location for an associated ring infrastructure resource, selectedfrom among the one or more ring infrastructure resources, is notoptimized on the ring of nodes. The determination that the hostinglocation is not optimized is based on one or more users of the componenthaving a common interest in the associated resource. It is indicatingthat at least one of the hosting location of and the access location forthe associated ring infrastructure resource on the ring of nodes is tobe optimized to reduce inter-node communication costs between nodes onthe ring nodes in response to the component request.

Embodiments of the present invention may comprise or utilize a specialpurpose or general-purpose computer including computer hardware, suchas, for example, one or more processors and system memory, as discussedin greater detail below. Embodiments within the scope of the presentinvention also include physical and other computer-readable media forcarrying or storing computer-executable instructions and/or datastructures. Such computer-readable media can be any available media thatcan be accessed by a general purpose or special purpose computer system.Computer-readable media that store computer-executable instructions arephysical storage media. Computer-readable media that carrycomputer-executable instructions are transmission media. Thus, by way ofexample, and not limitation, embodiments of the invention can compriseat least two distinctly different kinds of computer-readable media:computer storage media and transmission media.

Computer storage media includes RAM, ROM, EEPROM, CD-ROM or otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other medium which can be used to store desired programcode means in the form of computer-executable instructions or datastructures and which can be accessed by a general purpose or specialpurpose computer.

A “network” is defined as one or more data links that enable thetransport of electronic data between computer systems and/or modulesand/or other electronic devices. When information is transferred orprovided over a network or another communications connection (eitherhardwired, wireless, or a combination of hardwired or wireless) to acomputer, the computer properly views the connection as a transmissionmedium. Transmissions media can include a network and/or data linkswhich can be used to carry or desired program code means in the form ofcomputer-executable instructions or data structures and which can beaccessed by a general purpose or special purpose computer. Combinationsof the above should also be included within the scope ofcomputer-readable media.

Further, upon reaching various computer system components, program codemeans in the form of computer-executable instructions or data structurescan be transferred automatically from transmission media to computerstorage media (or vice versa). For example, computer-executableinstructions or data structures received over a network or data link canbe buffered in RAM within a network interface module (e.g., a “NIC”),and then eventually transferred to computer system RAM and/or to lessvolatile computer storage media at a computer system. Thus, it should beunderstood that computer storage media can be included in computersystem components that also (or even primarily) utilize transmissionmedia.

Computer-executable instructions comprise, for example, instructions anddata which, when executed at a processor, cause a general purposecomputer, special purpose computer, or special purpose processing deviceto perform a certain function or group of functions. The computerexecutable instructions may be, for example, binaries, intermediateformat instructions such as assembly language, or even source code.Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the described features or acts described above.Rather, the described features and acts are disclosed as example formsof implementing the claims.

Those skilled in the art will appreciate that the invention may bepracticed in network computing environments with many types of computersystem configurations, including, personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, mobile telephones,PDAs, pagers, routers, switches, and the like. The invention may also bepracticed in distributed system environments where local and remotecomputer systems, which are linked (either by hardwired data links,wireless data links, or by a combination of hardwired and wireless datalinks) through a network, both perform tasks. In a distributed systemenvironment, program modules may be located in both local and remotememory storage devices.

In this description and in the following claims, a “node” is defined asone or more software modules, one or more hardware modules, orcombinations thereof, that work together to perform operations onelectronic data. For example, the definition of a node includes thehardware components of a personal computer, as well as software modules,such as the operating system of the personal computer. The physicallayout of the modules is not important. A node can include one or morecomputers coupled via a network. Likewise, a node can include a singlephysical device (such as a mobile phone or Personal Digital Assistant“PDA”) where internal modules (such as a memory and processor) worktogether to perform operations on electronic data. Further, a node caninclude special purpose hardware, such as, for example, a router thatincludes special purpose integrated circuits.

Within this description and the following claims, a “resource” isdefined as electronic data that is consumable at, or may be acted uponby, a computer system, such as, for example, a client, a communicationintermediary, or a node. A non exhaustive list of examples of resourcesincludes files, devices, databases, data packets, electronic messages,data fields, as well as portions thereof. Resource is also defined toinclude identifiers within an identifier space, such as, for example,node identifiers used to identify nodes within a federationinfrastructure identifier space.

In this description and in the following claims, a “resource descriptor”is defined as a data structure (e.g., formatted in accordance with aresource descriptor schema) describing a resource.

In this description and in the following claims, a “namespace” isdefined as scoping mechanism for breaking up resources (e.g., all theresources on the Internet) into portions over which resolution,discovery, and message routing can be performed. Namespaces areextensible such that new scopes can be defined and individual scopes canbe hierarchical.

FIGS. 1A and 1B illustrate an example federation infrastructure 100 thatfacilitates providing optimized access to federationinfrastructure-based resources. Referring to FIG. 1, federationinfrastructure 100 includes ring 101 and client 102. Ring 101 and client102 can be connected to one another over (or is part of) a network, suchas, for example, a Local Area Network (“LAN”), a Wide Area Network(“WAN”), and even the Internet. Accordingly, ring 101 and client 102 aswell as any other connected computer systems and their components, cancreate message related data and exchange message related data (e.g.,Internet Protocol (“IP”) datagrams and other higher layer protocols thatutilize IP datagrams, such as, Transmission Control Protocol (“TCP”),Hypertext Transfer Protocol (“HTTP”), Simple Mail Transfer Protocol(“SMTP”), etc.) over the network.

As depicted in FIG. 1A, ring 101 includes a plurality of nodes, such as,for example, nodes 111-117, that have federated together based on avariety of possible federating partnerships. Generally, nodes canutilize federation protocols to form partnerships and exchangeinformation (e.g., state information related to interactions with othernodes). The formation of partnerships and exchange of informationfacilitates more efficient and reliable access to resources.

Other intermediary nodes (not shown) can exist before, after, and inbetween nodes 111-117. Thus, a message routed, for example, between node111 and node 116, can be pass through one or more of the otherintermediary nodes. In some embodiments, nodes are federated among oneanother as peers without a root node. Each node has a corresponding nodeID.

As described in further detail below, nodes can federate to formconsistent federations as well as best effort federations. Also asdescribed in further detail below, namespace management services can beused to organize resources into collections, potentially providingresource access through a plurality of namespaces.

Client 102 can be a computer system external to ring 101 that requestsaccess to resources hosted within ring 101. Client 102 can know how tocommunicate with one or more (but potentially not all) of the nodes ofring 101. Thus generally, client 102 can rely on federation protocolsused within ring 101 to have communication routed to an appropriatenode.

For example, client 102 may be aware of an entrée node into ring 101 aswell as a resource name of a resource within a namespace. As such,client 102 can submit the resource name to the entrée node. The entréenode can then utilize federation protocols within ring 101 to route amessage to the node hosting the named resource.

In some embodiments, nodes maintain processing information for providingoptimized access to federation infrastructure-based resources. Forexample, for a specified resource, a node on ring 101 can maintaininformation associating the specified resource with another node on ring101. The other node may be the node that hosts the specified resource.Alternately, the other node may be a node that, when contacted, resultsin reduced inter-ring communication cost to route a message to the nodehosting the specified resource. For example, requesting a resource froma node that has the hosting node for the resource in its neighborhoodset results in less inter-ring communication than requesting a resourcefrom a node that is not natively aware of the existence of the hostingnode (i.e., the node relies on federation protocols to make programtowards the hosting node).

A node can store various different types and instances of processinginformation. Processing information can include redirection information,class of service information, priority information, etc. How to optimizeany of the hosting location of a resource, access locations to aresource, and an appropriate entrée node for a request for a resourcecan determined based on available processing information in combinationwith characteristics of a requesting entity and/or a request.Characteristics of a requesting entity and/or a request include: clientcharacteristics, network diagnostic information, federationinfrastructure information, resource characteristics, applicationcharacteristics, date, time, etc. For example, a “premium” client may beredirected to a hosting node for a resource. However, other clients maybe redirected to a node some number of routing hops away from thehosting node for the resource.

The optimization algorithms used for optimization can be of arbitrarycomplexity. Algorithms can include various transforms that transform anincoming message into zero or more outgoing messages forwarded to entréenodes. For example, optimization algorithms can utilize processinginformation in combination with entity characteristics and/or requestcharacteristics to transform a destination address in a received messageinto zero or more different destination addresses for inclusion incorresponding outgoing messages.

As depicted in FIG. 1B, federation infrastructure 100 can also includecommunication intermediary 103. Communication intermediary 103 can be aproxy or other similar device that acts on behalf of external clients tocommunicate with ring 101. Communication intermediary 103 can forwardclient requests to nodes of ring 101. Communication intermediary 103 canalso return resources form ring 101 back to external clients.

In some embodiments, communication intermediary 103 communicates withring 101 in response to client requests. In other embodiments,communication intermediary 103 initiates communication with and/orreceives communicates from ring 101 (whether or not it has received anyclient requests). For example, nodes of ring 101 may from time to timepush processing information out to communication intermediary 103.Alternately, a request/reply mechanism can be used. Communicationintermediary 103 can send a request to a node on ring 101. In response,the ring can be return processing information back to communicationintermediary 103.

Communication intermediary 103 can maintain a cache of processinginformation for one or more specified resources. When a resource requestis received, communication intermediary 103 can check the cache todetermine if an optimized location for accessing the resource is in thecached processing information. When an optimized location is found,communication intermediary 103 can direct the client request to theoptimized location(s) as need. For example, if the request is notalready directed to the optimized location, communication intermediary103 can adjust the request to be sent to the optimized location. Whenadjusting a client request, communication intermediary 103 may or maynot notify a client that its request was adjusted. Further,communication intermediary 103 may or may not indicate the utilizedprocessing information to clients.

FIG. 4 illustrates a flow chart of an example method 400 for providingoptimized access to federation infrastructure-based resources. Method400 will be described with respect to the components and data ofcomputer architecture 100.

Method 400 includes an act of receiving a user request to access a ringinfrastructure resource, the user request sent from a requesting usercomponent (act 401). For example, referring to FIG. 1A, client 102 cansend resource request 131 to node 111. Node 111 can receive resourcerequest 131 from client 102. Resource request 131 can be a request toaccess resource 121 (e.g., a node identifier used to identify a nodewithin federation infrastructure 100). Resource request 131 can includea namespace path or other data identifying resource 121. A client canrequest access to a resource so that the client can take some action onthe resource, such as, for example, extracting data from, editing, ordeleting the resource.

Method 400 includes an act of detecting that the user request is notdirected to an optimized location for accessing the ring infrastructureresource within the ring infrastructure based on processing informationand one or more of: characteristics of the requesting user component andcharacteristics of the user request (act 402). For example, node 111 canrefer to stored processing information to determine that request 131 isnot directed to an optimized location to access resource 121 based onthe characteristics of client 102 and/or of request 131. As such, node111 can identify an appropriately optimized location for client 102 tocommunicate with (e.g., an appropriate entrée node) from the storedprocessing information. For example, node 111 can utilize variousoptimization algorithms to transform a destination address in request131 into zero or more different destination addresses for inclusion inother corresponding requests.

Processing information can be stored locally at a node and/or can bestored in some other location accessible to one or more nodes.

Method 400 includes an act of sending an indication of an appropriatelyoptimized location for the requesting user component to access the ringinfrastructure resource based on the processing information and the oneor more of the characteristics of the requesting user component and thecharacteristics of the user request (act 403). For example, node 111 cansend redirect notification 132 to client 102. Redirect notification 132can include the optimized location for accessing resource 121. Theoptimized location can be a destination address for inclusion in afurther request for resource 121.

In some embodiments, the appropriate optimized location for accessing aresource is the node that hosts the resource. For example, redirectnotification 132 can indicate that client 102 is to request access toresource 121 from node 116 (and can include a destination address fornode 116). In response to receiving redirect notification 132, client102 can send resource request 133 to node 116. Node 116 can receiveresource request 133 from client 102. Resource request 133 can be arequest to access resource 121. Resource request 133 can include anamespace path or other data identifying resource 121. Node 116 canrefer to stored processing information as well as one or more of:characteristics of client 102 and/or of request 133 to determine thatnode 116 is the optimized location to access resource 121. In response,node 116 can return access to resource 121 to client 102.

In other embodiments, the appropriate optimized location for accessing aresource is a node other than the node that hosts the resource. Forexample, redirect notification 132 can indicate that client 102 is torequest access to resource 121 from node 115 (and can include adestination address for node 115). In response to receiving redirectnotification 132, client 102 can send resource request 133 to node 115.Node 115 can receive resource request 133 from client 102. Node 115 canrefer to stored processing information as well as one or more of:characteristics of client 102 and/or of request 133 to determine thatnode 115 is the optimized location to access resource 121 based on thecharacteristics of client 102.

Accordingly, node 115 accepts resource request 133. Node 115 then usesinter-ring communication (e.g., federation protocols) to send furthermessages based on resource request 133 to node 116. Upon inter-ringcommunication reaching node 116, node 116 can return access to resource121 to client 102.

In further embodiments, redirect notification 132 can indicate thatclient 102 is to request access to resource 121 from some other node(e.g., node 191) of ring 101. In response to receiving redirectnotification 132, client 102 can send resource request 133 to the othernode. The other node can receive resource request 133 from client 102.The other node can refer to stored processing information (either storedlocally at the node or stored at some other location) as well as one ormore of: characteristics of client 102 and/or of request 133 todetermine that it is the optimized location to access resource 121.

Accordingly, the other node accepts resource request 133. The other nodethen uses inter-ring communication (e.g., federation protocols) to sendfurther messages based on resource request 133 to node 116. Theinter-ring communication can include using routing protocols to performone or more routing hops within ring 101. Upon inter-ring communicationreaching node 116, node 116 can return access to resource 121 to client102.

Referring now to FIG. 1B, client 102 can send resource request 141 tosome communications endpoint (e.g., a node on ring 101) based on anelectronic address. On the path to the communications endpoint, resourcerequest 141 can pass through communication intermediary 103. For examplewhen an IP packet is addressed to a given destination IP address, the IPpacket may pass through one or more routers (which are to some degreetransparent to each end of the end-to-end communication) along the pathto the communications endpoint. Resource request 141 can include anamespace path or other data identifying resource 121. From thenamespace path and/or other data, communication intermediary 103 candetermine that request 141 is a request to access resource 121.

Communication intermediary 103 can referred to cached processinginformation to determine if request 141 is directed to an appropriatelyoptimized location to access resource 121. When communicationintermediary 103 lacks processing information to make the determination,communication intermediary 103 can query a node of ring 101 for furtherprocessing information. From time to time, nodes of ring 101 can alsopush processing information to communication intermediary 103. Forexample, in response to a query or at a designate time, node 111 cansend mapping notification 142 to communication intermediary 103. Mappingnotification 142 can include processing information for determiningappropriately optimized locations for requesting access to resource 121.Push mechanisms can also be used in combination with other triggeringmechanisms for transferring processing information.

From mapping notification 142, communication intermediary 103 candetermine an appropriate optimized location for accessing resource 121.The appropriately optimized location can be node 116, node 115, or someother node. Communication intermediary 103 can then send resourcerequest 143 to the appropriately optimized location. Upon receivingrequest 143 or inter-ring communication representing request 143, node116 can return access to resource 121 back to communication intermediary103. Communication intermediary 103 can forward access to resource 121back to client 102. Alternately, node 116 can return access to resource121 directly back to client 102.

Redirection notifications and mapping notifications can be requestedand/or sent using request reply protocols, such as, for example,Hypertext Transfer Protocol (“HTTP”), or using other custom protocols.Components external to ring 101, such as, for example, client 102 and/orcommunication intermediary 103, can be external client-side hostedapplication/service components, external client-side federationinfrastructure components, or an underlying communications/routinginfrastructure transparent to a client. External communications/routinginfrastructure can be contained within a data center and/or may be anInternet overlay network, such as, for example, components within theInternet itself.

In some embodiments, a federation infrastructure includes a plurality ofdifferent layers providing different functions to a ring of nodes. FIG.2 illustrates an example of different layers within a federationinfrastructure 200. As depicted, federation infrastructure 200 includesring 201, naming service layer 202, communication service layer 203, andexternal client 204. Similar to ring 101, ring 201 can include aplurality of nodes that have federated together using one or morepossible federating partnerships. Naming service layer 202 convertsbetween namespace paths and node ids for nodes on ring 201.Communication service layer 203 implements communication abstractions toconvert between communication with external clients and inter-ringcommunication within ring 201. External client 204 can be any componentas previously described.

A component at any layer, including within ring 201, can signal alocation event change event to ring 201. For example, a federationinfrastructure membership component can signal location change events,including node Id ownership changed and node Id range ownership changed.A federation infrastructure storage component can signal location changeevents through the use of a ring dictionary data structure distributedacross the ring's nodes. A federation infrastructure naming service cansignal location change events for a given branch of a namespaceincluding node Id ownership changed and node Id range ownership changed.A federation infrastructure communication component can signal locationchange events for named groups implemented via a naming service. Theseand other location change events can be implemented in consistentfederations as well as in best effort federations.

A location change event can instruct ring 201 to adjust the hostinglocation and/or optimized access location for a resource. In response toa location change event, the hosting location of and/or optimized accesslocation for a resource can be changed. In response to a location changeevent, nodes within ring 201 can also update stored processinginformation for a resource. In some embodiments, stored processinginformation is updated even though a hosting location is not changed.For example, in response to a node failure or addition of a new node, anoptimized access location can change even when a resource remains hostedat the same location.

FIG. 3 illustrates an example federation infrastructure 300 thatfacilitates optimizing a hosting location for a federationinfrastructure-based resource. As depicted, federation infrastructure300 includes ring 301, naming service layer 302, and communicationservice layer 303. Component 304 resides at communication service layer303. Ring 301 includes nodes 311-317 along with one or more other nodesthat have federated together using one or more possible federatingpartnership.

FIG. 5 illustrates a flow chart of an example method 500 for configuringoptimized access to federation infrastructure-based resources. Method500 will be described with respect to the components and data offederation architecture 300.

Method 500 includes an act of detecting a component request to optimizeaccess to one or more ring infrastructure resources associated with thecomponent on the ring of nodes so as to reduce inter-node communicationbetween nodes on the ring of nodes (act 501). For example, component 304can send optimize message 311 to node 313. Optimize message 391 can be arequest to optimize access to resource 321. Node 313 can receiveoptimize message 391. Optimize message 391 is a request to optimizeaccess to one or more resources associated with ring 301, includingresource 321. Optimizing access to resource 321 can reduce inter-nodecommunication between nodes on ring 301.

Method 500 includes an act of determining that at least one of a hostinglocation of and an access location for an associated ring infrastructureresource, selected from among the one or more ring infrastructureresources, is not optimized on the ring of nodes based on one or moreusers of the component having a common interest in the associatedresource (act 502). For example, node 313 can determine that the hostinglocation and/or an access location for resource 321 is not optimized onring 301 based on one or more external clients having an interest inresource 321. A hosting location may not be optimized if another hostinglocation can be used to reduce inter-ring communication for accessingthe resource. An access location may not be optimized (for one or morecomponents) if another access location can be used to reduce inter-ringcommunication for accessing the resource. A non-optimized accesslocation can take into account various factors, such as, clientcharacteristics, network diagnostic information, federationinfrastructure information, resource characteristics, applicationcharacteristics, date, time, etc.

Method 500 includes an act of indicating that the at least one of thehosting location of and the access location for the associated ringinfrastructure resource is to be optimized on the ring of nodes toreduce inter-node communication costs between nodes on the ring nodes inresponse to the component request (act 503). For example, ring 301 canindicate that at least one of the host location and an access locationfor resource 321 is to be optimized. The host location of resource 321can then be moved from node 311 to node 315. Alternately or incombination, ring 301 can indicate that access locations for one or moreclients (not shown) that request access to resource 321 are also to beoptimized. Optimization of access locations can take into accountvarious factors, such as, client characteristics, requestcharacteristics, network diagnostic information, federationinfrastructure information, resource characteristics, applicationcharacteristics, date, time, etc.

In some embodiments, stateful and uniquely identifiable entities arehosted/owned (part of) by an application system (e.g., 303/304). Theentities can be associated with (e.g., interested in, having an affinityto, etc.) at least one resource in a federation infrastructure (e.g., inname service layer 302 or ring 301). For example, communication servicelayer 303 can maintain unique state, such as, for example, applicationcontext, for each of its external client connections that sourcing orsinking messages to/from a given name (i.e., the resource).Communication service layer 304 can indicate (via 301) that the uniquestate, or application context, is to be hosted optimally for at leastone resource. Naming service layer 302 can then, from time to time,indicate back up to communication service layer 303 that the uniquestate, or application context, is to be moved to a different node tomaintain appropriate cost with name service layer 302 and/or ring 30

Thus, an application system, such as, for example, communication servicelayer 303, can indicate (e.g., via optimization request 391) that atleast one of its application contexts has an affinity to at least oneresource (e.g., resource 321) hosted within federation infrastructure300. The resource can be hosted at another component of, or hosted by,federation infrastructure 300, such as, for example, at naming servicelayer 302 or ring 301. Subsequently, the lower layer (301 and/or 302)can source an event to the application system (303) indicating that theat least one application context is to be hosted on a different node.

Accordingly, embodiments of the invention include an overlay ring nodeAPI that indicates node ownership id changes. A hosted distributed ringdictionary rejects requests that are for a data element which thecurrent node is not the primary (owner).

A naming service can indicate to application context with an explicitnamespace graph-node affinity is to be hosted (in what node) in order tobe co-located with a subject name. At least one optimal location can beindicated as a result. Qualified forms of naming operations can berejected by naming service layer 302 if the application context is nothosted in an optimal machine. Accordingly, rejections indicate at leastone optimal hosting location.

In some embodiments, a group communications component (e.g., supportingcommunication service layer 303 and naming service layer 302 for namedgroup registry and discovery) signals communication service layer 303when a named-group communication endpoint is not optimally placed withinthe current host machine. A hosted application/service uses such asignal to further indicate to its external clients that furthercommunications must be redirected to another external communicationsendpoint. The external indication can be transferred using privateprotocol messages between the hosted application/service server-sidecomponent and its client-side component or, for example, using HTTPredirect responses.

As such, an application service hosted within the hosting federationinfrastructure is informed when ownership of a node id or range of nodeids changes. In response, the hosted application/service redirectsexternal traffic to the new owner ring node's machine of the subjectnode id or node id range. This can occur within a consistent or besteffort federation infrastructure.

Further, an application/service can be hosted within a federationinfrastructure that additionally provides a storage component thatsupports generation of location change-events to the hostedapplication/service when ownership of a subject data resource's hostingring node changes. Additionally, such a hosted application/service canuse these location change-events to redirect external client traffic tothe machine of the new owning ring node of the subject data resource.

An application/service can also be hosted within a hosting federationinfrastructure that provides a communications facility enabling thehosted application/service to establish and use named communicationgroups (possibly hierarchically) amongst multiple endpoints. Such acommunications facility can supporting various MEP such as 1:1, 1:n,n:1, and n:n. Thus, the communications facility can signal locationchange-events to the hosted application/service to optimize internalfederation infrastructure communication needed to maintain a given setof endpoints for a common named group. A hosted application/service alsouses these location change-events to redirect related external clienttraffic to at least one endpoint within a machine indicated in thelocation change-event received from the hosting federationinfrastructure.

In some embodiments, location change-events are sourced from the hostingfederation infrastructure resulting in corresponding signals, commands,or messages to be pushed into an external communications infrastructureinterconnecting the external client endpoints and federationinfrastructure and its hosted application/service effecting transparentredirection of client traffic.

Best Effort Federation Infrastructures

As previously described, in some embodiments nodes can federatedtogether to form a best effort federation infrastructure. FIG. 6illustrates an example of a federation infrastructure 600. Thefederation infrastructure 600 includes nodes 601, 602, 603, that canform different types of federating partnerships. For example, nodes 601,602, 603 can be federated among one another as peers without a rootnode. Each of nodes 601, 602, and 603 has a corresponding ID 671, 682,and 693 respectively.

Generally, the nodes 601, 602, 603, can utilize federation protocols toform partnerships and exchange information (e.g., state informationrelated to interactions with other nodes). The formation of partnershipsand exchange of information facilitates more efficient and reliableaccess to resources. Other intermediary nodes (not shown) can existbetween nodes 601, 602, and 603 (e.g., nodes having IDs between 671 and693). Thus, a message routed, for example, between node 601 and node603, can be pass through one or more of the other intermediary nodes.

Nodes in federation infrastructure 600 (including other intermediarynodes) can include corresponding rendezvous protocol stacks. Forexample, nodes 601, 602, and 603 include corresponding rendezvousprotocol stacks 641, 642, and 643 respectively. Each of the protocolsstacks 641, 642, and 643 includes an application layer (e.g.,application layers 621, 622, and 623) and other lower layers (e.g.,corresponding other lower layers 631, 632, and 633). Each layer in arendezvous protocol stack is responsible for different functionalityrelated to rendezvousing a resource request with a correspondingresource.

For example, other lower layers can include a channel layer, a routinglayer, and a function layer. Generally, a channel layer is responsiblefor reliably transporting a message (e.g., using WS-ReliableMessagingand Simple Object Access Protocol (“SOAP”)) from one endpoint to another(e.g., from node 601 to node 603). The channel layer is also responsiblefor processing incoming and outgoing reliable messaging headers andmaintaining state related to reliable messaging sessions.

Generally, a routing layer is responsible for computing the next hoptowards a destination. The routing layer is also responsible forprocessing incoming and outgoing addressing and routing message headersand maintaining routing state. Generally, a function layer isresponsible for issuing and processing rendezvous protocol messages,such as, join and depart requests, pings, updates, and other messages,as well as generation of responses to these messages. The function layerprocesses request messages from the routing layer and sends backcorresponding response messages, if any, to the originating node usingthe routing layer. The function layer also initiates request messagesand utilizes the routing layer to have the requests messages delivered.

Generally, an application layer processes non-rendezvous protocolspecific data delivered from the function layer (i.e., applicationmessages). The function layer can access application data from theapplication layer and get and put application data in rendezvousprotocol messages (e.g., pings and updates). That is, the function layercan cause application data to be piggybacked on rendezvous protocolmessages and can cause the application data to be passed back to theapplication layer in receiving rendezvous protocol nodes. In someembodiments, application data is used to identify resources and resourceinterests. Thus, an application layer can include application specificlogic and state that processes data received from and sent to the otherlower layers for purposes of identifying resources and resourceinterests.

Federating Mechanisms

Nodes can federate using a variety of different mechanisms. A firstfederating mechanism includes peer nodes forwarding information to allother peer nodes. When a node is to join a federation infrastructure,the node utilizes a broadcast/multicast discovery protocol, such as, forexample, WS-Discovery to announce its presence and issues abroadcast/multicast find to detect other nodes. The node thenestablishes a simple forwarding partnership with other nodes alreadypresent on the network and accepts new partnerships with newly joiningnodes. Thereafter, the node simply forwards all application specificmessages to all of its partner nodes.

A second federating mechanism includes peer nodes that most efficientlytransmit application specific messages to their destination(s). When anew node is to join a federation infrastructure, the new node utilizes abroadcast/multicast discovery protocol, such as, for example,WS-Discovery to announce its presence and issues a broadcast/multicastfind to detect other nodes that are part of the federationinfrastructure. Upon detecting another node, the new node establishes apartnership with the other node. From the established partnership, thenew node learns about the presence of other nodes already participatingin federation infrastructure. It then establishes partnerships withthese newly-learned nodes and accepts any new incoming partnershiprequests.

Both node arrivals/departures and registrations of interest in certainapplication specific messages are flooded through the federationinfrastructure resulting in every node having global knowledge of otherpartner nodes and registrations of interest in application specificmessages. With such global knowledge, any node can send applicationspecific messages directly to the nodes that have expressed interest inthe application specific message.

A third federating mechanism includes peer nodes indirectly forwardingall application specific messages to their destination/s. In this thirdmechanism, nodes are assigned identifiers (ID's), such as, for example,a 128-bit or 160-bit ID. The node responsible for a maintainingregistration of interest in a given application specific message can bedetermined to be the one whose ID is closest to the one obtained bymapping (e.g., hashing) the destination identity (e.g. URI) of theapplication specific message to this 128-bit or 160-bit ID-space.

In this third mechanism, node arrivals and departures are flooded overthe entire fabric. On the other hand, registrations of interest incertain application specific messages are forwarded to the nodesdetermined to be responsible for maintaining such registrationinformation. For scalability, load balancing, and fault-tolerance, thenode receiving registration of interest in certain application specificmessages can reliably flood that registration information within itsneighborhood set. The neighborhood set for a specified node can bedetermined to be the set of nodes having IDs within a predefined rangeon either side of the ID of specified node.

Similar to the second mechanism, a newly joining node utilizes abroadcast/multicast discovery protocol, such as, for example,WS-Discovery to announce its presence and issues a localbroadcast/multi-cast find to detect a node that is already part of thefederation infrastructure. The new node establishes a partnership withthe discovered node and uses that partnership to learn about thepresence of other new nodes participating in the federationinfrastructure. The new node then establishes further partnerships withthe newly discovered nodes and accepts any new incoming partnershiprequests. The new node accepts incoming registrations of interest incertain application layer specific resources from its partners for whichit is responsible and may flood them over its neighborhood set. Thus,messages can generally be forwarded to their final destination viaintermediary routing nodes (e.g., that a newly joining node haspartnered with or that a partner node is aware of).

In response to receiving an incoming application specific message, thenew node forwards the message to the partner node that may beresponsible for maintaining the registration information for thedestination specified in the message. Thus, when using this thirdmechanism, every node in the federation infrastructure has globalknowledge of all other nodes but the registration information isefficiently partitioned among the nodes. Application specific messagesare transmitted to their final destination via only the partner's nodesthat may have the responsibility for maintaining registrationinformation of interest in those application specific messages. Thus,indirection is accomplished by forwarding only to the partner node thathas global knowledge of the registration information of interest for themessage being processed. This is in contrast to the first mechanismwhere the indirection is accomplished by forwarding to all the partnernodes.

A fourth federating mechanism includes peer nodes that route messages toother peer nodes. This fourth mechanism differs from the third mechanismat least in that both node arrivals/departures and registrations ofinterest in certain application specific messages are all routed insteadbeing flooded. Routing protocols are designed to guarantee rendezvousbetween application specific messages and the registration messages thatexpress interest in those application specific messages.

FIG. 7 illustrates an example of a computer architecture 700 thatfacilitates routing requests indirectly to partners. Computerarchitecture 700 depicts different types of computer systems and devicespotentially spread across multiple local discovery scopes participatingin a federation infrastructure.

Workstation 733 can include a registered PnP provider instance. Toinform its partners of the presence of this PnP provider instance,workstation 733 routes registration request 701 over the federationinfrastructure. Registration request 701 is initially forwarded tolaptop 731, which in turn forwards registration request 701 to messagebroker 737, which in turn forwards registration request 701 to messagegateway 741. Message gateway 741 saves the registration informationregistration request 701 in its database and returns success message 704to workstation 733.

Subsequently, another registered provider instance, this time that ofrunning services, comes alive within the workstation 733. This time thenode is aware that message gateway 741 is responsible for registrationsand forwards registration request 705 to message gateway 741 directly.Message gateway 741 saves the registration information registrationrequest 705 in its database and returns success message 706 toworkstation 733.

Subsequently, the printer 736 (e.g., a UPnP printer) is powered on andsends announcement 707. Server 734 detects announcement 707 and routesregistration request 708 to message broker 737. Message broker 737forwards registration request 708 to message gateway 741. Messagegateway 741 saves the registration information registration request 708in its database and returns success message 791 to server 734.

Subsequently, personal computer 742 issues lookup request 792 todiscover all devices. Since personal computer 742 doesn't know where toforward lookup request 792, it routes lookup request 792 throughworkstation 743. As registration and lookup requests are routed to thesame destination, the routing protocol essentially guarantees rendezvousbetween the two requests resulting in workstation 743 forwards findrequest 792 to message gateway 741. Message gateway 741 looks up theregistration information maintained by it and forwards find request 792to both the workstation 733 and server 734. Workstation 733 and server734 send response messages 714 and 716 respectively to personal computer742.

This fourth mechanism works by routing (instead of flooding) a requestto the node (message gateway 741) that has global knowledge of theregistrations specified in a request. This fourth mechanism, as will bedescribed in further detail below, essentially guarantees that routingcan be accomplished in O(log N) hops, where N is the number of nodesparticipating in the federation infrastructure. Since this fourthmechanism efficiently partitions both node partnership and registrationinformation, it scales to very large networks, even the Internet.

Although a number of federating mechanisms have been described, it wouldbe apparent to one skilled in the art, after having reviewed thisdescription, that other federation mechanisms are possible.

Relationship Between Nodes in a Federation

Accordingly, a federation consists of a set of nodes that cooperateamong themselves to form a dynamic and scalable network in whichinformation can be systematically and efficiently disseminated andlocated. Nodes are organized to participate in a federation as a sortedlist using a binary relation that is reflexive, anti-symmetric,transitive, total, and defined over the domain of node identities. Bothends of the sorted list are joined, thereby forming a ring. Thus, eachnode in the list can view itself as being at the middle of the sortedlist (as a result of using modulo arithmetic). Further, the list isdoubly linked so that any node can traverse the list in eitherdirection.

Each federating node can be assigned an ID (e.g., by a random numbergenerator with duplicate detection) from a fixed set of IDs between 0and some fixed upper bound. Thus, adding 6 to an ID of the fixed upperbound results in an ID of zero (i.e., moving from the end of the linkedlist back to the beginning of the linked listed. In addition, a 1:1mapping function from the value domain of the node identities to thenodes themselves is defined.

FIG. 8 depicts an example linked list 804 and corresponding ring 806.Given such a ring, the following functions can be defined:

-   -   RouteNumerically(V, Msg): Given a value V from the value domain        of node identities and a message “Msg,” deliver the message to        node X whose identity can be mapped to V using the mapping        function.    -   Neighborhood(X, S): Neighborhood is the set of nodes on the        either side of node X with cardinality equal to S.

When every node in the federation has global knowledge of the ring,RouteNumerically(V, Msg) is implemented by directly sending Msg to thenode X, whose identity is obtained by applying the mapping function toV. Alternately, when nodes have limited knowledge of other nodes (e.g.,only of immediately adjacent nodes), RouteNumerically(V, Msg) isimplemented by forwarding the message to consecutive nodes along thering until it reaches the destination node X.

Alternately (and advantageously), nodes can store enough knowledge aboutthe ring to perform a distributed binary search (without having to haveglobal knowledge or implement routing between immediately adjacentnodes). The amount of ring knowledge is configurable such thatmaintaining the ring knowledge has a sufficiently small impact on eachnode but allows increased routing performance from the reduction in thenumber of routing hops.

As previously described, IDs can be assigned using the “<” (less than)relation defined over a sufficiently large, bounded set of naturalnumbers, meaning its range is over a finite set of numbers between 0 andsome fixed value, inclusive. Thus, every node participating in thefederation is assigned a natural number that lies between 0 and someappropriately-chosen upper bound, inclusive. The range does not have tobe tight and there can be gaps between numbers assigned to nodes. Thenumber assigned to a node serves as its identity in the ring. Themapping function accounts for gaps in the number space by mapping anumber falling in between two node identities to the node whose identityis numerically closest to the number.

This approach has a number of advantages. By assigning each node auniformly-distributed number, there is an increased likelihood that allsegments of the ring are uniformly populated. Further, successor,predecessor, and neighborhood computations can be done efficiently usingmodulo arithmetic.

In some embodiments, federating nodes are assigned an ID from within anID space so large that the chances of two nodes being assigned the sameID are highly unlikely (e.g., when random number generation is used).For example, a node can be assigned an ID in the range of 0 to b^(n)−6,where b equals, for example, 8 or 66 and n equals, for example, 628-bitor 660-bit equivalent digits. Accordingly, a node can be assigned an ID,for example, from a range of 0 to 66⁴⁰−6 (or approximately 6.461502E48).The range of 0 to 66⁴⁰−6 would provide, for example, a sufficient numberof IDs to assign every node on the Internet a unique ID.

Thus, each node in a federation can have:

-   -   An ID which is a numerical value uniformly distributed in the        range of 0 to b^(n)−1; and    -   A routing table consisting of (all arithmetic is done modulo        b^(n)):        -   Successor node (s);        -   Predecessor node (p);        -   Neighborhood nodes (p_(k), . . . , p₁, s, s₁, . . . , s_(j))            such that s_(j).s.id>(id+u/2), j≧v/2−1, and            p_(k).p.id<(id−u/2), and k≧v/2−1; and        -   Routing nodes (r_(−(n−1)), . . . , r⁻¹, r₁, . . . , r_(n−1))            such that r_(±1)=RouteNumerically(id±b^(i), Msg).            where b is the number base, n is the field size in number of            digits, u is the neighborhood range, v is the neighborhood            size, and the arithmetic is performed modulo b^(n). For good            routing efficiency and fault tolerance, values for u and v            can be u=b and v≧max(log₂(N), 4), where N is the total            number of nodes physically participating in the federation.            N can be estimated from the number of nodes present on a            ring segment whose length is greater than or equal to b, for            example, when there is a uniform distribution of IDs.            Typical values for b and n are b=8 or 66 and n=128-bit or            660-bit equivalent digits.

Accordingly, routing nodes can form a logarithmic index spanning a ring.Depending on the locations of nodes on a ring, a precise logarithmicindex is possible, for example, when there is an existing node at eachnumber in the set of id±b^(i) where i=(1, 2, . . . (n−1)). However, itmay be that there are not existing nodes at each number in the set. INthose cases, a node closest to id±b^(i) can be selected as a routingnode. The resulting logarithmic index is not precise and may even lackunique routing nodes for some numbers in the set.

Referring again to FIG. 8, FIG. 8 illustrates an example of a binaryrelation between nodes in a federation infrastructure in the form ofsorted list 804 and corresponding ring 806. The ID space of sorted list804 is in the range 0 to 2⁸−6 (or 255). That is, b=2 and n=8. Thus,nodes depicted in FIG. 8 are assigned IDs in a range from 0 to 255.Sorted list 804 utilizes a binary relation that is reflexive,anti-symmetric, transitive, total, and defined over the domain of nodeidentities. Both ends of sorted list 804 are joined, thereby formingring 806. This makes it possible for each node in FIG. 8 to view itselfas being at the middle of sorted list 804. The sorted list 804 is doublylinked so that any node can traverse the sorted list 804 in eitherdirection. Arithmetic for traversing sorted list 804 (or ring 806) isperformed modulo 2⁸. Thus, 255 (or the end of sorted list 804)+6=0 (orthe beginning of sorted list 804).

The routing table indicates that the successor to ID 64 is ID 76 (the IDimmediately clockwise from ID 64). The successor can change, forexample, when a new node (e.g., with an ID of 71) joins or an existingnode (e.g., ID 76) leaves the federation infrastructure. Likewise, therouting table indicates that the predecessor to ID 64 is ID 50 (the IDimmediately counters clockwise from ID 64). The predecessor can change,for example, when a new node (e.g., with an ID of 59) joins or anexisting node (e.g., ID 50) leaves the federation infrastructure.

The routing table further indicates that a set of neighborhood nodes toID 64 have IDs 83, 76, 50 and 46. A set of neighbor nodes can be aspecified number of nodes (i.e., neighborhood size v) that are within aspecified range (i.e., neighbor range u) of ID 64. A variety ofdifferent neighborhood sizes and neighbor ranges, such as, for example,V=4 and U=60, can potentially be used to identify the set ofneighborhood nodes. A neighborhood set can change, for example, whennodes join or leave the federation infrastructure or when the specifiednumber of nodes or specified range is changed.

The routing table further indicates that ID 64 can route to nodes havingIDs 200, 2, 30, 46, 50, 64, 64, 64, 64, 76, 83, 98, 635, and 200. Thislist is generated by identifying the node closest to each number in theset of id±2^(i) where i=(1, 2, 3, 4, 5, 6, 7). That is, b=2 and n=8. Forexample, the node having ID 76 can be identified from calculating theclosest node to 64+2³, or 72.

A node can route messages (e.g., requests for access to resources)directly to a predecessor node, a successor node, any node in a set ofneighborhood nodes, or any routing node. In some embodiments, nodesimplement a numeric routing function to route messages. Thus,RouteNumerically(V, Msg) can be implemented at node X to deliver Msg tothe node Y in the federation whose ID is numerically closest to V, andreturn node Y's ID to node X. For example, the node having ID 64 canimplement RouteNumerically(243, Msg) to cause a message to be routed tothe node having ID 250. However, since ID 250 is not a routing node forID 64, ID 64 can route the message to ID 2 (the closest routing node to243). The node having ID 2 can in turn implement RouteNumerically(243,Msg) to cause the message to be routed (directly or through furtherintermediary nodes) to the node having ID 250. Thus, it may be that aRouteNumerically function is recursively invoked with each invocationrouting a message closer to the destination.

Proximity

Advantageously, other embodiments of the present invention facilitatepartitioning a ring into a ring of rings or tree of rings based on aplurality of proximity criteria of one or more proximity categories(e.g., geographical boundaries, routing characteristics (e.g., IProuting hops), administrative domains, organizational boundaries, etc.).It should be understood a ring can be partitioned more than once usingthe same type of proximity criteria. For example, a ring can bepartition based on a continent proximity criteria and a countryproximity criteria (both of a geographical boundaries proximitycategory).

Since IDs can be uniformly distributed across an ID space (a result ofrandom number generation) there is a high probability that any givensegment of a circular ID space contains nodes that belong to differentproximity classes provided those classes have approximately the samecardinality. The probability increases further when there are asufficient number of nodes to obtain meaningful statistical behavior.

Thus, neighborhood nodes of any given node are typically well dispersedfrom the proximality point of view. Since published application statecan be replicated among neighborhood nodes, the published informationcan be well dispersed as well from the proximality point of view.

FIG. 9 illustrates a ring of rings 900 that facilitates proximalrouting. Ring 901 can be viewed as a master or root ring, and containsall the nodes in each of the rings 902, 903, and 904. Each of the rings902, 903, and 904 contain a subset of nodes from ring 901 that arepartitioned based on a specified proximity criterion. For example, ring901 may be partitioned based on geographic location, where ring 902contains nodes in North America, ring 903 contains nodes in Europe, andring 904 contains nodes in Asia.

In a numerical space containing 65,536 (2¹⁶) IDs, routing a message froma North American node having an ID 5,345 to an Asian node having an ID23,345 can include routing the message within ring 902 until a neighbornode of the Asian node is identified. The neighbor node can then routethe message to the Asian node. Thus, a single hop (as opposed tomultiple hops) is made between a North American node and an Asian node.Accordingly, routing is performed in a resource efficient manner.

FIG. 10 illustrates an example proximity induced partition tree of rings1000 that facilitates proximal routing. As depicted, partition tree ofrings 1000 includes a number of rings. Each of the rings represents apartition of a sorted linked list. Each ring including a plurality anodes having IDs in the sorted linked list. However for clarity due tothe number of potential nodes, the nodes are not expressly depicted onthe rings (e.g., the ID space of partition tree 1000 may be b=66 andn=40).

Within partition tree 1000, root ring 1001 is partitioned into aplurality of sub-rings, including sub-rings 1011, 1012, 1013, and 1014,based on criterion 1071 (a first administrative domain boundarycriterion). For example, each component of a DNS name can be considereda proximity criterion with the partial order among them induced pertheir order of appearance in the DNS name read right to left.Accordingly, sub-ring 1011 can be further partitioned into a pluralityof sub-rings, including sub-rings 1021, 1022, and 1023, based oncriterion 1081 (a second administrative domain boundary criterion).

Sub-ring 1022 can be further partitioned into a plurality of sub-rings,including sub-rings 1031, 1032, and 1033, based on criterion 1072 (ageographic boundary criterion). Location based proximity criterion canbe partially ordered along the lines of continents, countries, postalzip codes, and so on. Postal zip codes are themselves hierarchicallyorganized meaning that they can be seen as further inducing a partiallyordered sub-list of proximity criteria.

Sub-ring 1031 can be further partitioned into a plurality of sub-rings,including sub-rings 1041, 1042, 1043, and 1044, based on criterion 1073(a first organizational boundary criterion). A partially ordered list ofproximity criterion can be induced along the lines of how a givencompany is organizationally structured such as divisions, departments,and product groups. Accordingly, sub-ring 1043 can be furtherpartitioned into a plurality of sub-rings, including sub-rings 1051 and1052, based on criterion 1083 (a second organizational boundarycriterion).

Within partition tree 1000, each node has a single ID and participatesin rings along a corresponding partition path starting from the root toa leaf. For example, each node participating in sub-ring 1052 would alsoparticipate in sub-rings 1043, 1031, 1022, 1011 and in root 1001.Routing to a destination node (ID) can be accomplished by implementing aRouteProximally function, as follows:

-   -   RouteProximally(V, Msg, P): Given a value V from the domain of        node identities and a message “Msg,” deliver the message to the        node Y whose identity can be mapped to V among the nodes        considered equivalent by the proximity criteria P.

Thus, routing can be accomplished by progressively moving closer to thedestination node within a given ring until no further progress can bemade by routing within that ring as determined from the condition thatthe destination node lies between the current node and its successor orpredecessor node. At this point, the current node starts routing via itspartner nodes in the next larger ring in which it participates. Thisprocess of progressively moving towards the destination node by climbingalong the partitioning path towards the root ring terminates when theclosest node to the destination node is reached within the requestedproximal context, as originally specified in the RouteProximallyinvocation.

Routing hops can remain in the proximal neighborhood of the node thatoriginated the request until no further progress can be made within thatneighborhood because the destination node exists outside it. At thispoint, the proximity criterion is relaxed to increase the size of theproximal neighborhood to make further progress. This process is repeateduntil the proximal neighborhood is sufficiently expanded to include thedestination node (ID). The routing hop made after each successiverelaxation of proximal neighborhood criterion can be a potentiallylarger jump in proximal space while making a correspondingly smallerjump in the numerical space compared to the previous hop. Thus, only theabsolutely required number of such (inter-ring) hops is made before thedestination is reached.

It may be the case that some hops are avoided for lookup messages sincepublished application data gets replicated down the partition tree whenit is replicated among the neighborhood nodes of the destination node.

To accomplish proximal routing, each federation node maintainsreferences to its successor and predecessor nodes in all the rings itparticipates as a member (similar to successor and predecessor for asingle ring)—the proximal predecessor, proximal successor, and proximalneighborhood. In order to make the routing efficient, the nodes can alsomaintain reference to other nodes closest to an exponentially increasingdistance on its either half of the ring as routing partners (similar torouting nodes for a single ring). In some embodiments, routing partnernodes that lie between a pair of consecutive successor or predecessornodes participate in the same lowest ring shared by the current node andthe node numerically closest to it among the successor or predecessornode pairs respectively. Thus, routing hops towards a destination nodetransition into using a relaxed proximity criterion (i.e., transitioningto a higher ring) only when absolutely needed to make further progress.Accordingly, messages can be efficiently rendezvoused with acorresponding federation node.

In some embodiments, nodes implement a proximal routing function toroute messages based on equivalence criteria relations. Thus, given anumber V and a message “Msg”, a node can implement RouteProximally(V,Msg, P) to deliver the message to the node Y whose identify can bemapped to V among the nodes considered equivalent by proximity criterionP. The proximity criterion P identifies the lowest ring in the partitiontree that is the common ancestor to all the nodes considered proximallyequivalent by it. It can be represented as a string obtained byconcatenating the proximity criterion found along the path from the rootring to the ring identified by it separated by the path separatorcharacter ‘/’. For example, the proximity criterion identifying sub-ring1042 can be represented as “Proximity:/.COM/Corp2/LocationA/Div2”. Eachring in the partition tree 1000 can be assigned a unique number, forexample, by hashing its representational string with a SHA basedalgorithm. If the number 0 is reserved for the root ring, it can beinferred that RouteNumerically(V, Msg)=RouteProximally(V, Msg, 0).

For example, a node in sub-ring 1044 can implement RouteProximally toidentify a closer node in sub-ring 1031 (e.g., to a node in sub-ring1013). In turn, sub-ring 1031 can implement RouteProximally to identifya closer node in sub-ring 1022. Likewise, sub-ring 1022 can implementRouteProximally to identify a closer node in sub-ring 1011. Similarly,sub-ring 1011 can implement RouteProximally to identify a closer node inring 1001. Thus, it may be that a RouteProximally function isrecursively invoked with each invocation routing a message closer to thedestination.

Thus, when proximity criterion is taken into account, routing hops on apath to a final destination can remain within the proximity of a nodethat originates a request, while making significant progress between theoriginating node and the destination node in a numerical space, untileither the destination node is reached or no further progress can bemade under the chosen proximity criterion at which point it is relaxedjust enough to make further progress towards the destination. Forexample, proximity criterion can be relaxed enough for a message to berouted from ring 1031 up to ring 1022, etc.

Utilizing the above approach to proximity, it is possible to confinepublished information to a given ring. For example, organizations maylike to ensure that organization specific information is not availableto entities outside of their trust domains either (1) implicitly in theform of neighborhood replication to nodes outside of their domains or(2) explicitly in the form of servicing lookup requests for suchinformation. The first aspect is satisfied by replicating publishedinformation only among the nodes neighboring the target ID within thespecified ring. Because all messages originated by a node are routed bysuccessively climbing the rings to which it belongs towards the rootring, there is a high likelihood that all lookup requests originatedwithin an organization will be able to locate the published informationconfined to it thereby implicitly satisfying the second aspect.

Also, organizations dislike nodes automatically federating with nodesoutside of their trust domain. This can happen, for example, when avisiting sales person connects his/her laptop computer to the network inthe customer premises. Ideally, the laptop computer belonging to thesales person wishes to locate information published in its home domainand/or federate with the nodes in its home domain starting at its lowestpreferred proximity ring. It will typically not be permitted to federatewith the nodes in the customer's domain. Supporting this scenariorequires ability to locate seed nodes in the home domain. Such seednodes can be used for locating information published in the home domain,to join the home federation, to selectively import and export publishedinformation across domains, and as one possible way to arbitrateconflicting failure reports submitted by other nodes. Seed nodes arealso sometimes referred as message gateways.

In other embodiments, an entity publishes references to seed nodes inthe root ring. Seed nodes can be published at the unique number (such asthe one obtained by hashing its representational string) associated withthe ring (as a target ID). Seed node information can further beon-demand cached by the nodes in various rings that are on the path tothe corresponding target IDs in the root ring. Such on-demand cachingprovides for improved performance and reduction in hotspots that mightoccur when semi-static information is looked up quite frequently. Seednode information can also be obtained via other means such as DNS.

To provide fault tolerance for confined published information, each nodecan maintain a set of neighborhood nodes in all of the rings itparticipates in. Given the above, the state maintained by a node can besummarized as follows:

-   -   An ID which is a numerical value uniformly distributed in the        range of 0 to b^(n)−1.    -   A routing table consisting of (all arithmetic is done modulo        b^(n)):        -   For each ring, say ring d, in which the node participates            -   Successor node (s_(d))            -   Predecessor node (p_(d))            -   Neighborhood nodes (p_(kd), . . . , p_(1d), p_(d),                s_(d), s_(1d), . . . , s_(jd)) such that                s_(jd).s_(d).id>(id+u/2), j>v/2−1,                p_(kd).p_(d).id<(id−u/2), and k≧v/2−1.        -   Routing nodes (r_(−(n−1)), r⁻¹, r₁, r_(n−1)) such that            r_(=i)=RouteProximally(id±b^(i), updateMsg, d) such that            s_(d)≦id+b^(i)≦s_(d+1) or p_(d+1)≦id−b^(i)≦p_(d) as            appropriate.    -   where b is the number base, n is the field size in number of        digits, u is the neighborhood range, and v is the neighborhood        size.

Note that a subset of the neighborhood nodes maintained by a given nodein ring “d” can appear again as neighborhood nodes in the child ring“d+1” in which the given node participates as well. As such one canderive the upper bound on the total number of neighborhood nodesmaintained by a given node across all the D rings it participates asD*max(u,v)/2. This considers that only one reference to a given node iskept and the worst case upper bound is for a balanced tree.

It should be noted that when a ring is partitioned into a plurality ofcorresponding sibling sub-rings, it is permitted for a specified node tosimultaneously participate in more than one of the plurality ofcorresponding sibling sub-rings, for example, through aliasing. Aliasingcan be implemented to associate different state, for example, fromdifferent sub-rings, with the specified node. Thus, although aliases fora given node have the same ID, each alias can have distinct stateassociated with them. Aliasing allows the specified node to participatein multiple rings having distinct proximity criteria that are notnecessarily common ancestors of more specific proximity criteria. Thatis, the specified node can participate in multiple branches of theproximity tree.

For example, a dual NIC (wired and wireless) laptop can be considered tobe proximally equivalent to both other wireless and wired nodes sharingthe same LAN segments as the laptop. But, these two distinct proximitycriteria can be modeled as sub-criteria that are applicable only afterapplication of a different higher priority proximity criterion, such as,for example, one based on organizational membership. As the laptopbelongs to the same organization, the aliased nodes in the two sub-ringsrepresenting 1) membership in the wired and 2) membership in thewireless LAN segments merge into a single node in the ring representingthe organization to which the laptop belongs. It should be understandthat the RouteProximally works as expected without any modifications inthe presence of aliasing.

Each proximal ring can be configured in accordance with (potentiallydifferent) ring parameters. Ring parameters can be used to define aneighborhood (e.g., ring parameters can represent a neighborhood range,a neighborhood size, ping message and depart message timing anddistribution patterns for ping and depart messages), indicate aparticular federating mechanisms (e.g., from among the above-describedfirst through fourth federating mechanisms previously described or fromamong other federating mechanisms), or define communication specificsbetween routing partners in the same proximal ring. Some ring parametersmay be more general, applying to a plurality of different federatingmechanisms, while other ring parameters are more specific and apply tospecific type of federating mechanism.

Ring parameters used to configure a higher level proximal ring can beinherited in some embodiments by lower level proximal rings. Forexample, it may be that ring 1043 inherits some of the ring parametersof ring 1031 (which in turn inherited from ring 1022, etc.). Thus, aneighborhood size and neighborhood range associated with ring 1031 isalso associated with ring 1041.

However, inherited ring parameters can be altered and/or proximal ringscan be individually configured in accordance with different ringparameters. For example, it may be that ring 1011 is for anadministrative domain that contains a large number of nodes and thus theabove-described fourth federating mechanism is more appropriate for ring1011. On the other hand, it may be that ring 1021 is for a smallbusiness with a relatively smaller number of nodes and thus theabove-described second federating mechanism is more appropriate for ring1021. Thus, the ring parameters associated with ring 1021 can be set to(or inherited parameters changed to) different values than the ringparameters associated with ring 1011. For example, a ring parameterindicating a particular type of federating mechanisms can be differentbetween rings 1011 and 1021. Similarly parameters defining aneighborhood can be different between rings 1011 and 1021. Further, ring1021 can be configured in accordance with specific parameters that arespecific to the above-described second federating mechanism, while ring1011 is configured in accordance additional with specific parametersthat are specific to the above-described fourth federating mechanism.

Accordingly, proximal rings can be flexibly configured based on thecharacteristics (e.g., number, included resources, etc.) of nodes in theproximal rings. For example, an administrator can select ring parametersfor proximal rings using a configuration procedure (e.g., through auser-interface). A configuration procedure can facilitate theconfiguration of inheritance relationships between proximal rings aswell as the configuration of individual proximal rings, such as, forexample, to override otherwise inherited ring parameters.

Embodiments of the invention include partitioning nodes of a federationinfrastructure. Partitioning nodes of a federation infrastructure caninclude an act of accessing a sorted linked list containing node IDsthat have been assigned to nodes in a federation infrastructure Forexample, referring back to FIG. 10, a sorted linked list represented byring 1001 can be accessed. The node IDs of the sorted linked list (thenodes depicted on ring 1001) can represent nodes in a federationinfrastructure (e.g., federation infrastructure 600).

Partitioning nodes of a federation infrastructure can include an act ofaccessing proximity categories that represent a plurality of differentproximity criteria for partitioning the sorted linked list. For example,proximity criterion representing domain boundaries 1061, geographicalboundaries 1062, and organizational boundaries 1063 can be accessed.However, other proximity criteria, such as, trust domain boundaries, canalso be represented in accessed proximity criterion. Proximitycategories can include previously created partially ordered lists ofproximity criteria. A ring can be partitioned based on partially orderedlists of proximity criteria.

Partitioning nodes of a federation infrastructure can include an act ofpartitioning the sorted link list into one or more first sub lists basedon a first proximity criterion, each of the one or more first sub listscontaining at least a subset of the node IDs from the sorted linkedlist. For example, ring 1001 can be partitioned into sub-rings 1011,1012, 1013, and 1014 based on criterion 1071. Each of sub-rings 1011,1012, 1013, and 1014 can contain a different sub-set of node IDs fromring 1001.

Partitioning nodes of a federation infrastructure can include an act ofpartitioning a first sub list, selected from among the one or more firstsub lists, into one or more second sub lists based on a second proximitycriterion, each of the one or more second sub lists containing at leasta subset of node IDs contained in the first sub list. For example,sub-ring 1011 can be partitioned into sub-rings 1021, 1022, and 1023based on criterion 1081. Each of the sub-rings 1021, 1022, and 1023 cancontain a different sub-set of node IDs from sub-ring 1011.

Embodiments of the invention include populating a node's routing table.Populating a node's routing table can include an act of inserting apredecessor node into a routing table, the predecessor node preceding acurrent node relative to the current node in a first direction of asorted linked list. For example, referring to FIG. 8 the node having ID50 can be inserted into the routing table as a predecessor for the nodehaving ID 64 (the current node). Moving in a clockwise direction 821(from end A of sorted linked list 804 towards end B of sorted linkedlist 804), the node having ID 50 precedes the node having ID 64.Inserting a predecessor node can establish a symmetric partnershipbetween the current node and the predecessor node such that current nodeis a partner of predecessor node and the predecessor node is a partnerof the current node.

Populating a node's routing table can include an act of inserting asuccessor node into the routing table, the successor node succeeding thecurrent node relative to the current node in the first direction in thesorted linked list. For example, the node having ID 76 can be insertedinto the routing table as a successor for the node having ID 64 (thecurrent node). Moving in a counter-clockwise direction 822, the nodehaving ID 76 succeeds the node having ID 64. Inserting a successor nodecan establish a symmetric partnership between the current node and thesuccessor node such that current node is a partner of the successor nodeand the successor node is a partner of the current node.

Populating a node's routing table can include an act of insertingappropriate neighborhood nodes into the routing table, the neighborhoodnodes identified from the sorted linked list in both the first directionand in a second opposite direction based on a neighborhood range andneighborhood size. For example, the nodes having IDs 83, 76, 50, and 46can be inserted into the routing table as neighborhood nodes for thenode having ID 64 (the current node). Based on a neighborhood range of20 and a neighborhood size 4, the nodes having IDs 83 and 76 can beidentified in clockwise direction 821 and the nodes having IDs 50 and 46can be identified in counter-clockwise direction 822 (moving from end Bof sorted linked list 304 towards end A of sorted linked list 804). Itmay be that in some environments no appropriate neighborhood nodes areidentified. Inserting a neighborhood node can establish a symmetricpartnership between the current node and the neighborhood node such thatcurrent node is a partner of the neighborhood node and the neighborhoodnode is a partner of the current node.

Populating a node's routing table can include an act of insertingappropriate routing nodes into the routing table, the routing nodesidentified from the sorted linked list in both the first and seconddirections based on the a number base and field size of the ID space forthe federation infrastructure, the routing nodes representing alogarithmic index of the sorted link list in both the first and seconddirections. For example, the nodes having IDs 200, 2, 30, 46, 50, 64,64, 64, 64, 64, 76, 83, 98, 135 and 200 can be inserted into the routingtable as routing nodes for the node having ID 64. Based on the numberbase 2 and field size of 8 the nodes having IDs 64, 64, 76, 83, 98, 135and 200 can be identified in direction 821 and the nodes having IDs 64,64, 50, 46, 30, 2, and 200 can be identified in direction 822. Asdepicted inside ring 806, the routing nodes represent a logarithmicindex of the sorted link list 804 in both clockwise direction 821 andcounter-clockwise direction 822. Inserting a routing node can establisha symmetric partnership between the current node and the routing nodesuch that current node is a partner of the routing node and the routingnode is a partner of the current node.

Embodiments of the invention include populating a node routing tablethat takes proximity criteria into account. Populating a node routingtable that takes proximity criteria into account can include an act ofinserting a predecessor node for each hierarchically partitioned routingring the current node participates in into a routing table. Eachpredecessor node precedes the current node in a first direction (e.g.,clockwise) within each hierarchically partitioned routing ring thecurrent node participates in. The hierarchically partitioned routingrings are partitioned in accordance with corresponding proximitycriteria and contain at least subsets of a bi-directionally linked list(and possibly the whole bi-directionally linked list). For example,referring again to FIG. 10, it may be that a specified node participatesin root ring 1001 and sub-rings 1011, 1022, 1023, 1031, and 1042. Thus,a predecessor node is selected for the specified node from within eachof the rings 1001 and sub-rings 1011, 1022, 1023, 1031, and 1042.

Populating a node routing table that takes proximity criteria intoaccount can include an act of inserting a successor node for eachhierarchically partitioned routing ring the current node participates ininto the routing table. Each successor node succeeding the current nodein the first direction within each hierarchically partitioned routingring the current node participates in. For example, a successor node isselected for the specified node from within each of the rings 1001 andsub-rings 1011, 1022, 1023, 1031, and 1042.

Populating a node routing table that takes proximity criteria intoaccount can include an act of inserting appropriate neighborhood nodesfor each hierarchically partitioned routing ring the current nodeparticipates in into the routing table. The neighborhood nodes can beidentified in both the first direction (e.g., clockwise) and in a secondopposite direction (e.g., counter clockwise) based on a neighborhoodrange and neighborhood size from the hierarchically partitioned routingrings the current node participates in. For example, neighborhood nodescan be identified for the specified node from within each of the rings1001 and sub-rings 1011, 1022, 1023, 1031, and 1042.

Populating a node routing table that takes proximity criteria intoaccount can include an act of inserting appropriate routing nodes foreach hierarchically partitioned routing ring the current nodeparticipates in into the routing table. For example, routing nodes canbe identified for the specified node from within each of the rings 1001and sub-rings 1011, 1022, 1023, 1031, and 1042.

In some embodiments, appropriate routing nodes are inserted for eachproximity ring d except the leaf ring (or leaf rings in embodiments thatutilize aliasing), in which the node Y participates. Appropriate routingnodes can be inserted based on the following expression(s):if Y.s _(d).id<Y.id+b ^(i) <Y.s _(d+1).id is true, then use ring d; orif Y.p _(d).id<Y.id−b ^(i) <Y.p _(d+1).id is true, then use ring d.

If a ring has not been identified in the previous step, use the lead(e.g., ring 501) ring as ring d. Now, ring d is the proximity ring inwhich node Y should look for the routing partner closest to z.

Embodiments of the invention include routing a message towards adestination node. Routing a message towards a destination node caninclude an act of a receiving node receiving a message along with anumber indicating a destination. For example, referring again to FIG. 8,the node having ID 64 can receive a message indicating a destination of212.

Routing a message towards a destination node can include an act ofdetermining that the receiving node is at least one of numericallyfurther from the destination than a corresponding predecessor node andnumerically further from the destination than a corresponding successornode. For example, in direction 822, ID 64 is further from destination212 than ID 50 and, in direction 821, ID 64 is further from destination212 than ID 76. Routing a message towards a destination node can includean act of determining that the destination is not within a neighborhoodset of nodes corresponding to the receiving node. For example, the nodewith ID 64 can determine that destination 212 is not within theneighborhood set of 83, 76, 50, and 46.

Routing a message towards a destination node can include an act ofidentifying an intermediate node from a routing table corresponding tothe receiving node, the intermediate node being numerically closer tothe destination than other routing nodes in the corresponding routingtable. For example, the node having ID 64 can identify the routing nodehaving ID 200 as being numerically closer to destination 212 that otherrouting nodes. Routing a message towards a destination node can includean act of sending the message to the intermediate node. For example, thenode having ID 64 can send the message to the node having ID 200.

Embodiments of the invention include routing a message towards adestination node based on proximity criteria. Routing a message towardsa destination node based on proximity criteria can include an act of areceiving node receiving a message along with a number indicating adestination and a proximity criterion. The proximity criterion definesone or more classes of nodes. The receiving node receives the message aspart of a current class of nodes selected from among the one or moreclasses of nodes based on the proximity criterion. For example,referring to FIG. 9, the node having ID 172 can receive a messageindicating a destination of 201 and proximity criterion indicating thatthe destination node be part of classes represented by ring 901. Thenode having ID 172 can receive the message as part of ring 904.

Routing a message towards a destination node based on proximity criteriacan include an act of determining that the receiving node is at leastone of, numerically further from the destination than a correspondingpredecessor node and numerically further from the destination than acorresponding successor node, among nodes in a selected class of nodes.For example, within ring 904, the node with ID 172 is further fromdestination 201 than the node having ID 174 in the clockwise directionand is further from destination 201 than the node having ID 153 in thecounterclockwise direction.

Routing a message towards a destination node based on proximity criteriacan include an act of determining that the destination is not within thereceiving node's neighborhood set of nodes for any of the one or moreclasses of nodes defined by the proximity criterion. For example, thenode having ID 172 can determine that destination 201 is not in acorresponding neighborhood set in ring 904 or in ring 901.

Routing a message towards a destination node based on proximity criteriacan include an act of identifying an intermediate node from thereceiving node's routing table, the intermediate node being numericallycloser to the destination than other routing nodes in the routing table.For example, the node having ID 172 can identify the node having ID 194as being numerically closer to destination 201 than other routing nodesin ring 904. Routing a message towards a destination node based onproximity criteria can include an act of sending the message to theintermediate node. For example, the node having ID 172 can send thereceived message to the node having ID 194. The node having ID 172 cansend the received message to the node having ID 194 to honor apreviously defined partially ordered list of proximity criterion.

Node 194 may be as close to destination 201 as is possible within ring904. Thus, proximity can be relaxed just enough to enable furtherrouting towards the destination to be made in ring 901 in the next leg.That is, routing is transitioned from ring 904 to ring 901 since nofurther progress towards the destination can be made on ring 904.Alternately, it may be that the node having ID 201 is within theneighborhood of the node having ID 194 in ring 901 resulting in nofurther routing. Thus, in some embodiments, relaxing proximity criteriato get to the next higher ring is enough to cause further routing.

However, in other embodiments, incremental relaxation of proximitycriteria causing transition to the next higher ring continues untilfurther routing can occur (or until the root ring is encountered). Thatis, a plurality of transitions to higher rings occurs before furtherrouting progress can be made. For example, referring now to FIG. 10,when no further routing progress can be made on ring 1031, proximitycriteria may be relaxed enough to transition to ring 1011 or even toroot ring 1001.

Node Phases

A node participating in a federation infrastructure can operate indifferent operational phases. Valid phase values for a node can bedefined to be members of an ordered set. For example,{NodeId}.{InstanceIds}. {Phase Value [Phase-State Values: Inserting,Syncing, Routing, Operating]. [Phase.Unknown Indication: phase known attime of transmission, phase unknown at time of transmission]} definesone possible ordered set representing a phase-space of a given nodewithin a federation infrastructure. A node instance can transition (oradvance) through the node phase-states from Inserting to Syncing toRouting to Operating in order. Further, in some embodiments, a nodeinstance can be configured such that the node instance is prevented fromtransitioning back to a prior node phase-state. In some embodiments, anode advances its instance ID each time the node comes up.

For example, a node instance can prevented from transitioning fromRouting back to Syncing (or back to Inserting), etc. Accordingly, insome embodiments, when it is known that a given node instance (e.g.,identified by (NodeId, InstanceId)) has advanced to a particular nodephase-state (e.g., Operating), it is also known that the given nodeinstance is not likely to (and in some embodiments will not) revert to aprior node phase-state (e.g., back to Routing, Syncing, or Inserting).Thus, there is a significant likelihood that any node instance in a nodephase prior to the particular node phase-state is a new (and advanced)instance of the node.

In some embodiments, phase information and corresponding instance Ids(which advance as a node comes up) are transferred together. Thus, it ispossible to determine that a lesser node phase-state for the sameinstance is older. Further, when a newer node instance is known (at anyphase-state values) any information about older instances is consideredout of date.

From time to time, nodes can reboot or lose communication with oneanother, such as, for example, when first starting up, through agraceful departure, or as a result of abnormal termination (crash).Thus, there is the potential for a node in any node phase-state toreboot or lose communication with other nodes. For example, a crash cancause a node in a Routing phase-state to reboot. During a reboot or loseof communication, there may be no way to determine what node phase-statea node is in. Accordingly, when a node is rebooting or communication toa node is lost, a [Phase Unknown Indication] can be set to indicate thatthe phase-state for the node is currently not known. However, anypreviously expressed and/or detected phase-state for the node can bemaintained and is not lost.

The [Phase Unknown Indication] can be used to indicate whether aphase-state was known at the time a phase-state value was transmitted(e.g phase value with phase unknown not set) or if a phase-state is apreviously expressed phase-state and the phase-state was not known atthe time the phase-state was transmitted (e.g., phase value withphase.unknown set). Thus, the phase of a node (its phase value) can berepresented using both a phase-state value and a phase.unknownindication.

Join Protocol

From time to time, nodes can join to and depart from existingfederations. The nodes can implement appropriate protocols for joiningand departing federations. For example, a node can implement a Join( )function to become part of an existing federation. A node implementingthe Join( ) function can transition through three ordered phase-states:an inserting phase-state, a synchronizing phase-state, and a routingphase-state before reaching the final operating phase-state. In otherembodiments these specific order phase-states may not exist while othersmay be defined. FIG. 11A illustrates an example of a node establishingmembership within a federation infrastructure. FIG. 11B illustrates anexample of nodes in a federation infrastructure exchanging messages.

Insertion Phase:

A node, Y, enters this phase-state by issuing a join message, includingat least its node ID and indicating a join action to the federation. Ajoin message can be a routed message sent by a newly joining node (nodeY) with its destination property set to the identity of the newlyjoining node. In this phase-state, a newly joining node is insertedbetween its predecessor and successor nodes in the federation. Theinsertion phase-state can be implemented according to the followingalgorithm (All arithmetic is performed modulo b^(n)):

IP1 Y identifies an existing node that is already part of a lowest ringfrom which the joining node wishes to participate in the federation.This can either be statically configured or dynamically discovered usingDHCP and/or DNS and/or WS-Discovery or a (potentially well-known)constant. Let this existing federation node be E.

IP2. Y invokes E.RouteNumerically(Y, joinMsg) to determine the node Xwhose ID is numerically closest to Y.id in every proximity ring that thenode Y participates. This can include routing a join message to multiplenodes.

IP3. Determine the numerical successor (s) and predecessor (p) nodes.(Note that the data needed to do the following insertion can be carriedin the join message and its response. As such, there are no additionalroundtrips needed.)

Case 1: X.id>Y.id

-   -   Y.s=X, Y.p=X.p, X.p.s=Y, and X.p=Y

Case 2: X.id<Y.id

-   -   Y.p=X, Y.s=X.s, X.s.p=Y, and X.s=Y

In response to the join message, node X (the node that processed thejoin message) can send a join response back to node Y. The join responsecan indicate the predecessor node (Y.p) and successor node (Y.s) fornode Y. Node Y can receive the join response and process the joinresponse to become aware of its predecessor and successor nodes. Afterprocessing the join response, Node Y can be a weak routing participantin the federation. For example, Node Y can simply forward message sentto it, either to its successor or predecessor nodes. Thus, Node Y isinserted into the federation infrastructure but routing and neighborhoodtables are not populated. Before reaching this point, node Y willrequest other nodes sending it messages to redirect the messages sent toit through a different node by returning a status message to the sendingnode indicating that node Y's liveness phase is in an insertingphase-state.

Generally, from time to time, nodes can exchange sync request andresponse messages. Sync request and sync response messages can includeliveness information (e.g., headers) for other nodes from the sender'spoint of view. Neighborhood state can also be included in sync requestand response messages such that application layers in a neighborhood areaware of one another's state. One example of when sync request andresponse messages are exchanged is during a synchronizing phase-state ofa joining node. However, sync request and response messages can beexchanged during other operational phase-states as well (e.g. while inthe Operating Phase-state).

FIG. 12 depicts an example of a message model and related processingmodel 1200. As depicted in FIG. 12, a node can send and receive syncrequests messages. For example, sync request message 1201 can bereceived at function layer 1651 from a newly inserted node (e.g., thenode in FIG. 11B having ID 144). Application data 1202 (e.g., namespacesubscriptions) can be piggybacked in sync request message 1201. Functionlayer 1251 can inform application layer 1252 of any application dataincluded in sync requests messages. For example, function layer 1251 caninvoke neighborhood state sync event 1203, including application data1202, to application layer 1252. Sync request 1231, includingapplication data 1207, can also be sent to another node that processessync request 1231 similar to the processing to sync request 1201 inprocessing model 1200.

In response to some function layer event (e.g., sync request message1201, sync response message 1241, or ping message 1212) function layer1251 can invoke the neighborhood state request function 1204 inapplication layer 1252. Neighborhood state request 1204 is a request tothe application layer to obtain the state that needs to be propagated inthe neighborhood. In response to neighborhood state request 1204,application layer 1252 can supply neighborhood state 1206, includingoptional application data 1207, to function layer 1251. Alternately,application layer 1252 can send neighborhood state 1206, includingoptional application data 1207 in reaction to some application layerevent. Using internal mechanisms similar to the above, function layer1251 can send sync response message 1208, including optional applicationdata 1207, to propagate application layer neighborhood state.

Synchronization Phase:

After processing a join response message, a node Y transitions from theinsertion phase-state to synchronizing (Syncing) phase-state. In thesynchronization phase-state, the newly-inserted node Y synchronizesinformation with nodes in the neighborhood. Generally, Node Y can sendsync messages to at least its predecessor and successor nodes identifiedin the insertion phase-state. These nodes processing the sync messagescan return sync responses that indicate corresponding neighborhood androuting partner nodes of these processing nodes. In a more specificexample, the synchronizing phase-state can be implemented according tothe following algorithm (All arithmetic is performed modulo b^(n)):

SP1. Compute the Neighborhood(Y) from the union of Neighborhood(Y.s) andNeighborhood(Y.p) nodes in each proximal ring the node Y participates.The union computation can be done as follows:(s _(j) , . . . , s ₁ , s, p, p _(i) , . . . , pk) such that s _(j).s.id>(Y.id+u/2), j≧v/2−1, p _(k) .p.id<(Y.id−u/2), and k>v/2−1

SP2. Referring briefly to FIG. 12, query Y's local application layer(e.g., application layer 1252) via a neighborhood state request (e.g.,neighborhood state request) 1204 to obtain optional application specificneighborhood data (e.g., application specific data 1207).

SP3. Send synchronize message to at least the proximal successor andpredecessor nodes including at least liveness state information of eachproximal neighborhood and routing partner node from Y's perspective. Anyoptional application specific neighborhood data (e.g., application data1207) accessed via SP 2 is included in the sync request 1231.

SP3. Y receives sync response messages back from those nodes processingsync messages sent in SP2. For example, node Y can exchange synchronizemessages (request/response) with one or more nodes within its computedneighborhood. After synchronize messages are exchanged with at least oneand potentially all of a node Y's neighborhood nodes, the computedneighborhood nodes can exchange further messages to propagatesynchronized data. A synchronization message (request or response) canbe a non-routed message sent by a node to proactively synchronize itsdata with a target node that is, for example, in the nodes neighborhood.

SP4. As sync response message in SP3 are received (e.g., sync responsemessage 1241), any optional application specific neighborhood datapresent in these received sync response messages (e.g., application data1222) can be offered to Y's application layer 1252 via neighborhoodstate sync event 1203.

As part of the synchronizing phase-state, the proximal successor (e.g.,Y.s) and predecessor (Y.p) nodes exchange their routing tables with thenewly-inserted node (e.g., Y). Nodes that receive sync messages canrespond by sending sync responses. Sync responses carry data similar tosynchronize messages except from the perspective of the responding node.Both sync messages and sync responses can carry (or piggyback)application data. Thus, application data can be propagated between nodesduring the synchronizing phase-state. When the synchronize phase-stateis complete, the node can process messages destined for it, instead ofsimply forwarding them either to a successor or predecessor. However,the node may still be viewed as a weak routing participant because itsrouting table is not populated.

Routing Phase:

After the synchronizing phase-state is completed, a node transitionsinto the routing phase-state. In the routing phase-state, thenewly-synchronized node (e.g., node Y) computes its routing nodes. Therouting phase-state can be implemented according to the followingalgorithm (All arithmetic is performed modulo b^(n)):

RP1 If the routing phase-state is being executed as part of thebalancing procedure (explained later), ensure that the successor node(Y.s) and the predecessor node (Y.p) are alive in every proximity ringthe node Y participates. If either is not alive, determine thereplacement node for the failed one(s) by choosing a next best successoror predecessor node among the neighborhood nodes in the ring underconsideration.

RP2. For 1≦i≦n−1

-   -   RP2 a. Compute z=Y.id±b^(i)    -   RP2 b. If the ring d is not the most specific proximity, find        the proximity ring d in which the node Y participates and        satisfying the condition Y.s_(d).id<Y.id+b^(i)<Y.s_(d)+1.id or        Y.p_(d).id<Y.id−b^(i)<Y.p_(d+1).id. Else make ring d the most        specific proximity ring. Ring d is the proximity ring in which        node Y should look for the routing partner closest to z. Let Q        be the node numerically closest to z between and Y.s_(d).r_(±i)        and Y.p_(d).r_(±i). If |Q.id-z| is within a configurable        percentage of b^(i) (typically 20%), simply make If Q.id is        closer to z than either (Y.s_(d).id±b^(i)) or        (Y.p_(d).id±b^(i)), it means node Y is a better partner routing        node to node Q in proximity ring d than either Y.s_(d) or        Y.p_(d). Therefore, send updateMsg to node Q, if it has not        already been sent, supplying i and node Y as parameters so that        node Q can establish node Y as its partner routing node at        r_(−i).    -   RP2 c. If this phase-state is being executed as part of the        balancing procedure and if Y.s_(d).r_(±i).id==Y.p_(d).r_(±i).id,        there is only one node in the numerical range between        (Y.s_(d).id±b^(i)) and (Y.p_(d).id±b^(i)). That node is the one        pointed to by the routing node r_(±i) of the successor (or        predecessor) node. Therefore, simply make        Y.r_(±i)=Y.s_(d).r_(±i,i).    -   RP2 d. Else, compute the routing partner Y.r_(±i) by invoking        RouteProximally on node Q with the proximity criterion set to        that of ring d. This implies Y.r_(±i)=Q.RouteProximally(z,        updateMsg, d).

RP3. At this point, node Y can process not only messages destined for itbut can also route messages.

RP4. Subscribe to liveness notification events sent from the applicationlayer for the endpoint IDs of the partner routing nodes, if this has notalready been done. Also, revoke any liveness event subscriptionspreviously established with the application layer for the nodes that areno longer partner routing nodes. For example, referring briefly back toFIG. 6, subscription and/or revoke requests can be passed up to anapplication layer (e.g., application layer 621) that implements pub-sublogic for a corresponding application (e.g., a namespace application).When subsequent application specific liveness messages (e.g. thoseresulting from namespace subscriptions) are received at the applicationlayer, notifications (events) can be pushed down to other lower layers(e.g., other lower layers 631) for processing.

FIG. 13 depicts an example of a number of liveness interactions that canoccur between function layer 1351 and application layer 1352. Asdepicted in FIG. 13, endpoints are, for example, publish/subscribetopics (e.g., represented by a URL or URI) representing various nodesand can be, for example, federation infrastructure nodes. Subscribe ToLiveness Event 1301 can be invoked from function layer 1351 toapplication layer 1352 to subscribe to a liveness event (e.g., to apublish/subscribe topic). Revoke Liveness Subscription 1302 can beinvoked from function layer 1351 to application layer 1352 to revoke asubscription to a liveness event. End Point Down Event 1303 can be sentfrom application layer 1352 to function layer 1351 to indicate that anendpoint may be down and provide function layer 1351 with an optionalreplacement endpoint. End Point Down Event 1303 can be sentasynchronously based on a prior subscription (e.g., Subscribe ToLiveness Event 1301).

Node Down 1304 can be invoked from function layer 1351 to applicationlayer 1352 to indicate that function layer 1351 (or some other lowerlayer) has detected a failed node and optionally provide applicationlayer 1352 with a replacement node. Application layer 1352 cansubsequently propagate that a potentially failed node was detected toother interested parties. Node down event 1304 can be sentasynchronously anytime function layer 1351 or some other lower layerdetects a potentially failed node. Send liveness 1306 can be invokedfrom application layer 1352 to function layer 1351 when applicationlayer 1352 detects that a node is down (e.g., from node down event 1304or from some other out-of-band mechanism). Send liveness event 1306 cancause function layer 1351 to send a liveness message. Send livenessevent 1306 can also be invoked asynchronously anytime application layer1352 detects that a node is down and does not depend on any priorestablished subscriptions (via subscribe to liveness).

Thus, in some embodiments, function layer 1351 is used recursively. Forexample, function layer 1351 can indicate an interest in a specifiednode (e.g., is the particular node up or down) to application layer1352. Application layer 1352 can formulate an application specificsubscription for notifications related to the specified node and thenreuse function layer 1351 to communicate the formulated subscription toappropriate corresponding application layer 1352 instances in otherfederation nodes. For example if the application layers 1352 with infederation nodes implemented a namespaces pub/sub behaviors, functionlayer 1351 can route the subscription to a publish/subscribe managerthat manages notifications for the specified node—the pub/sub Managerbeing implemented as at least part of the application 1352 in therelated federation nodes. Accordingly, function layer 1351 is used toroute a subscription that function layer 1351 caused to be generated.Similar recursive mechanisms can also be used to unsubscribe orotherwise indicate that there is no longer an interest in the specifiednode.

Operating Phase:

After the routing phase-state is completed, a node transitions into theoperating phase-state. The node can remain in an operating phase-stateuntil it goes down (e.g., rebooting). In the operating phase-state, thenode can send update messages to routing partners from time to time.Update messages (both update requests and update responses) can includeneighborhood node liveness information for the sending nodes (e.g., forall proximal neighborhoods of interest). This sent liveness informationcan also include that of the sender's liveness info. Update messages canbe routed messages originated by nodes to periodically update itsrouting partner nodes. Application data can be piggyback on updatemessages such that application data can be propagated during routingpartner updates. The message destination is set to the identity of theperfect routing partner at the desired routing index. The Message IDproperty of this message is assigned an application sequence number soas to enable the node(s) processing this message to determine the latestmessage and this message is routed proximally.

A node that receives an update message can respond with an updateresponse. An update response carries the same data as the update messageexcept that the data is from the perspective of the responding node.Through the exchange of update messages and update responses nodes canexchange routing information. From time to time, operational nodes canupdate routing partners.

From time to time, operational nodes can also send ping messages (e.g.,ping messages 1209 and 1211 in FIG. 12). A ping message is a one-waymessage sent by a node to periodically announce its presence anddisseminate information within its neighborhood about itsneighborhood/routing nodes and replicate (e.g., piggybacked) applicationdata.

An origin node can send a ping message to one or more of its immediatepredecessor and successor neighborhood nodes. Thus, depending on theping distribution pattern (i.e., which nodes are sent ping messages)information related to the origin node is propagated to other nodes on aring within the neighborhood of the origin node. For example, the originnode can send a ping message only to its immediate predecessor andsuccessor nodes and the ping message propagates outward from theposition (node ID) of the origin node along the ring in both directionsto the edge of the origin's neighborhood. Alternately, the origin nodecan send a ping message to every n^(th) node in its neighborhood in bothits predecessor and successor directions.

Each node receiving a ping message checks its interest in the originnode from a neighborhood range perspective. If not interested, itdiscards the ping message. If interested it processes the ping messageand forwards the ping message according to its specified ping pattern ifsuch forwarding is constrained to the neighborhood of the originatingnode. For example, after processing a ping message a receiving node canforward the ping message to at least its successor node if the sendingand origin nodes are in its predecessor node set or at least itspredecessor node if the sending and origin node are in its successorset.

Thus, the outward propagation of ping messages stops when the messagereaches the edge of the neighborhood node set around the origin node.The Message ID property of ping message is assigned an applicationsequence number so as to enable the nodes processing this message todetermine the latest message from the origin node and avoid duplicateprocessing or otherwise unneeded forwarding.

Referring back to FIG. 12, ping message 1209 can be received at functionlayer 1251 from a neighborhood node. Application data 1212 (e.g.,namespace subscriptions) can be piggybacked in ping message 1209.Function layer 1251 can inform application layer 1252 of any applicationdata included in ping messages. Similarly, function layer 1251 caninform application layer 1252 of any application data included in SyncRequest messages. Both of these cases of transference can beaccomplished via sending a neighborhood state sync event 1203, includingapplication data 1212, to application layer 1252.

In response to some function layer event (e.g., received ping message1209) function layer 1251 can send neighborhood state request 1204 toapplication layer 1252. Neighborhood state request 1204 is invoked onthe application layer 1252 to obtain the state that needs to beoptionally propagated in the neighborhood. In response to neighborhoodstate request 1204, application layer 1252 can return neighborhood state1206, including optional application data 1207, to function layer 1251.Function layer 1251 can send ping message 1211, including optionalapplication data 1207, to propagate neighborhood and routing partnernode liveness information as well as optional application layerneighborhood state. Function layer 1251 can also send sync response1208, including optional application data 1207, to propagate applicationstate.

Departure Protocol

When it is appropriate for a node to depart from a federation, the nodecan implement a Depart function to be gracefully removed from thefederation. A node departs an existing federation by sending a departuremessage to one or more of its immediate proximal predecessor andsuccessor nodes, and maybe other nodes in the same proximalneighborhood. Thus, depending on the departure distribution pattern(i.e., which nodes are sent departure messages) information related tothe departing node is propagated to other nodes on a ring within theneighborhood of the departing node. A departure message is a one-waymessage originated by a gracefully departing node to inform one or moreother nodes within at least one of its proximal neighborhoods about itsimpending departure. The departing node propagates the depart message(e.g., within its neighborhood) in a manner similar to the propagationof the ping messages. For example, referring back to FIG. 11, the nodehaving ID 30 can send depart messages 1119 to the nodes having IDs 17and 40. The node having ID 30 can then remove itself from the federationinfrastructure from the standpoint of a given proximal ring. Note thatit is possible that a node remove itself from one proximal neighborhoodbut not others to which it may belong.

Since the nodes having IDs 17 and 40 (i.e., the predecessor andsuccessor nodes) are likely to be the closest nodes to ID 30 after thenode having ID 30 is removed, the nodes having IDs 17 and 40 are madeaware of the node having ID 30's departure. Thus, future messages thatare to be delivered to ID 30 can be appropriately processed at the nodeshaving IDs 17 and 40. The nodes having IDs 17 and 40 can propagate thedeparture of the node having ID 30 to the other nodes on ring 1206. Inthe absence of the node having ID 30, the nodes have IDs 17 and 40 canalso recompute predecessor and successor pointers, potentially pointingto each other.

The Message ID property of a depart message is assigned the sameapplication sequence ID as that of Ping messages so as to enable thenodes processing the depart message to determine the latest messageamong a series of ping and depart messages sent by an origin node.Graceful departure from a federation proximal ring is optional butencouraged. However, the federation is designed to self-heal if nodesleave abruptly.

Liveness

During the lifetime of a federation, nodes can exchange livenessinformation to maintain the federation. Liveness information can beincluded in virtually any message that is exchanged within a federationin the form of Liveness Message Headers. For example, join messages,join responses, sync messages, sync responses, update messages, updateresponse, application specific messages, liveness messages, and pingmessages can all include liveness information headers. When a federationnode sends any message or response, the node can include Livenessinformation for processing by other nodes. Liveness information can beincluded in a liveness information header of liveness message.

Liveness information indicating the liveness state of a node can berepresented using the following properties:

-   -   [Node]: Identifies the node whose liveness state is being        represented. A node can be identified based on [Reference        Properties] that further include an [Instance ID].        -   [Reference Properties]: Element information items specified            in the WS-addressing specification. WS-addressing defines            the [Instance ID] reference property for inclusion in the            reference property set.            -   [Instance ID]: A number that identifies a particular                instance of a node. An incrementing boot count can be                used as the instance ID of a node.    -   [Phase]: Conveys the phase of identified node.        -   [Phase-State Value] Conveys the highest phase-state            (inserting, synchronizing, routing, operating) that the            indicated node instance was know to have achieved        -   [Phase Unknown Indication] An indicator that conveys if the            current phase is known or unknown.    -   [Freshness]: Conveys the freshness of the information and its        value ranges from 0 to MaxFreshness. The higher the value, the        fresher the information with 0 implying no information and        MaxFreshness is a protocol defined constant.    -   [Color]: Identifies the proximity equivalence class to which the        node belongs. Two nodes with the same color value are always        considered to be proximally closest because they both belong to        the same equivalence class identified by the color value. The        number of proximity equivalence classes can increase over time        as more nodes join the federation.    -   [Weight]: Supplies the node capability metric and its value        ranges from 0 to MaxWeight. It measures the desirable        characteristics of a federation node such as large computational        power, high network bandwidth, and long uptime. The higher the        value, the more capable the node is making it more desirable        from a partnership perspective.

In some environments, the [Node] and [Freshness] properties of a nodeare either implicitly or explicitly conveyed in a larger scope such asthe [Origin] and [Sender] message headers and as such inclusion of theabove properties again in the liveness headers will be duplicative. Forexample the sender of a message need only convey its current phase,color, and weight information as its ID, Instance Id are supplied in themessage addressing headers and its Freshness is implied.

Liveness state can be at least partially ordered based on a “<” binaryrelation defined as follows:

“L1<L2” is true if

-   -   1. “L1.[Node].[Name]=L2.[Node].[Name]” is true and one of the        following is true with the tests performed and short-circuited        in the order listed:        -   L1.[Node].[Reference Properties].[Instance            ID]<L2.[Node].[Reference Properties].[Instance ID]        -   L1.[Phase.Unknown Indication]!=true AND L2.[Phase Unknown            Indication]!=true AND L1.[Phase-State]<L2.[Phase-State]        -   L1.[Freshness]<L2.[Freshness]    -   2. Or “L1.[Color]=L2.[Color]” is true and one of the following        is true with the tests performed and short-circuited in the        order listed:        -   L1.[Phase-State]<L2.[Phase-State]        -   L1.[Weight]<L2.[Weight]

Further, a liveness “down” message can be sent to a specified node whenit is detected or suspected that the specified node has becomeunavailable (e.g. gone down). As an example, referring briefly againback to FIG. 6, when an application layer (e.g., application layer 621)detects that another application layer (e.g., application layer 623) ora node hosting that another application layer is down, the detectingapplication layer can notify other lower layers (e.g., other lowerlayers 631) that the node may be down, for example, in accordance withmessage model and related processing models 1200 and/or 1300. Such anotification can cause other lower layers, such as, for example,function layer 1251, to send a liveness down message. This is only oneexample of stimulus for the generation of liveness down messages.

Since liveness down messages are routed and thus delivered to a nodeclosest to those nodes suspected of being down, if a liveness downmessage for a specified node gets delivered back to the specified node,then either the specified node never went down or the specified node isa different instance (e.g., with a different instance ID). On the otherhand, if the liveness down message gets delivered to another node, itindicates the specified node does appear to have gone down. Accordingly,if the node receiving the liveness down message views itself as being inthe proximal neighborhood of the specified node, it may source adeparture message for the specified node into that proximal neighborhoodas described as well as indicating to its the application layer (e.g.,using Node Down 1304) that the specified node may be down and that thereceiving node is its replacement. A liveness down message for thespecified node can be routed proximally with its target ID set to thatof the node that may be down.

Balancing Procedure

Embodiments of the present invention are designed to accommodate largenumber of nodes joining and departing the federation in a short periodof time. Such changes in the network can cause routing delays if thelogarithmic search trees maintained at the various nodes becomeunbalanced. That is, if there are more nodes on one side of a ring thanthe other. To facilitate optimal routing efficiency, nodes participatingin a federation execute the balancing procedure when certain criteriaare met.

For example, when any of the following conditions are true, any node canexecute the balancing procedure to ensure a balanced routing table foroptimal routing efficiency:

-   -   A configured number of liveness messages described above were        received.    -   A configured amount of time has elapsed since the receipt of the        last liveness message described above.    -   The neighborhood has changed in the sense that some new nodes        have arrived or some existing nodes have departed.

Balancing the routing tables is a simple process. For example, nodeswith an unbalanced routing table can re-execute the Synchronization andRouting phase-states of the Join protocol.

Acts RP2 b, RP2 d and RP4 combined with 1) finding the closest routingnode to a number, 2) the departure protocol followed by the nodesleaving a federation gracefully, and 3) balancing procedure followed bythe nodes receiving liveness messages result in a the faster healingsystem when federating nodes join and depart the network fairly quicklyand in large numbers.

Status Messages

A status message is non-routed message sent by a receiver node to asender node to inform routing success/failure of a correlated messagethat the sender node previously forwarded to the receiver node. FIG. 14depicts an example of how messages forming part of a request-responsemessage exchange pattern are routed across nodes on a ring. A statusmessage can include headers that identify the original correlatedmessage whose routing status is being reported. As such, status messagescan be used between nodes to indicate that message was successfullyrouted form one node to the next. For example, routing request message1411 from node 1401 to node 1406 includes sending request 1411 thoughnodes 1402, 1403, 1404, and 1405. Corresponding cascading success statusmessages (status 1417, 1418, 1419, 1420 and 1421) can be sent from node1406 to node 1405, from node 1405 to node 1804, from node 1404 to node1403, from mode 1403 to node 1402, and from node 1402 to node 1401respectively. In response to request 1411, response 1416 can be sentend-to-end from node 1407 to node 1401. Response 1416 is optional andmay not exist in a one-way message exchange pattern.

Joining A Federation Infrastructure

Embodiments of the invention include a node joining a federationinfrastructure. Joining a federation infrastructure can include an actof issuing a join message to a federation infrastructure. For example,referring to Figure back to FIG. 11A, the node having ID 144 can issuejoin 1101 to federation infrastructure including ring 1106. Joining afederation infrastructure can include an act of receiving a join messagefrom a joining node. For example, an existing node in the federationinfrastructure including ring 1106 can receive join 1101.

Joining a federation infrastructure can include an act of routing a joinmessage to a processing node. The processing node can be a node havingan ID numerically closer the ID of the joining node than other activenodes in the federation infrastructure at the time the join message isbeing routed. For example, join 1101 can initially be received at thenode having ID 64, routed to the node having ID 135 and routing to thenode having ID 151.

Joining a federation infrastructure can include an act of computing oneor more predecessor nodes and one or more successor nodes for thejoining node. For example, the node having ID 151 can compute animmediate predecessor node and an immediate successor node for the nodehaving ID 144. Within ring 1106, the node having ID 151 can compute thatthe node having ID 135 is an immediate predecessor node that the nodehaving ID 151 is an immediate successor node. Similar computations canbe made for other proximal rings.

Joining a federation infrastructure can include an act of computing oneor more routing nodes for the joining node. For example, the node havingID 151 can compute routing nodes (from the node having ID 151'sperspective) for the node having ID 144. Within ring 1106, the nodehaving ID 151 can compute, for example, that the nodes having IDs 218and 40 are routing nodes for the node having ID 144. Similarcomputations can be made for other proximal rings.

Joining a federation infrastructure can include an act of sending a joinresponse to the joining node. A join response can identify all thepredecessor and successor neighborhood and routing partner nodes for thejoining node as computed by the processing node given its current viewof the federation infrastructure. For example, join response 1102 canidentify at least the node having ID 135 as the immediate predecessornode to the node have ID 144, can identify the node having ID 151 as theimmediate successor node to the node having ID 144, and can identify anyrouting nodes (for the node having ID 144) computed at the node havingID 151 for node ID 144 (the newly joining node).

Joining a federation infrastructure can include an act of receiving ajoin response from a federation node that processed the join message.For example, the node having ID 144 can receive join response 1102 fromthe node having ID 151.

Joining a federation infrastructure can include an act of sending a syncrequest to at least each of the immediate proximal predecessor nodes andimmediate proximal successor nodes. For example, referring now to FIG.11B, the node having ID 144 can send sync requests 1103 to the nodeshaving IDs 135 and 151. Sync request 1103 can include an identificationof any neighborhood nodes of the node having ID 144 and/or anidentification of any routing partners of the node having ID 144.

The nodes having IDs 135 and 151 can receive the sync requests 1103. Inresponse to receiving sync requests 1103, the nodes having IDs 135 and151 can identify their neighborhood and routing partner nodes fromcorresponding routing tables. The nodes having IDs 135 and 151 caninclude their identified neighborhood and routing partner nodes'liveness information in sync response 1104 and send the send syncresponses 1104 to the node having ID 144.

Joining a federation infrastructure can include an act of receiving async response from each of the proximal predecessor and successor nodes.For example, the node having ID 144 can receive sync responses 1104 fromthe nodes having IDs 135 and 151. Sync response 1104 can includeliveness information for one or more nodes on ring 1106 or other ringsin a federation infrastructure. Sync response 1104 can also identify anyprospective routing partner nodes for the node having ID 144.

Joining a federation infrastructure can include an act of computingneighbor nodes. For example, the node having ID 144 can computecorresponding neighborhood nodes based on the union of the neighborhoodnodes for the nodes having IDs 135 and 151. Neighborhood nodes can becomputed based on a summarized view of the join response message and anysync response messages.

Joining a federation infrastructure can include an act of computingrouting nodes. For example, the node having ID 144 can compute routingnodes from among the nodes of ring 1106. Routing partners can becomputed base on a summarized view of the join response message and anysync response messages.

Joining a federation infrastructure can include an act of exchanging atleast neighborhood node information with computed routing partners. Forexample, the node having ID 144 and the node having ID 218 (a computedrouting partner) can exchange state information (e.g., instance ID,phase-state, etc) corresponding to their respective neighborhood nodes.These exchanges are accomplished by the newly joining node sourcing(routing) an Update message to at least each unique computed routingpartner as described in the Routing Phase-state text above. The nodesprocessing the Update message will send corresponding Update responsemessage in reaction to the receipt of these update messages from thenewly joining node. The Update response includes at least the livenessinformation for itself and its neighborhood nodes.

Joining a federation infrastructure can also include an act ofinitiating an initial propagation of routing tables to at least oneneighborhood node. For example, the node having ID 144 can includecomputed neighborhood and routing partner nodes in a ping message andsend the ping message to the node having ID 174 (e.g., one of thecomputed neighborhood nodes). The node having ID 174 can receive theping message and update a corresponding routing table with the livenessinformation originated at the node having ID 144. The node having ID 174can also include its corresponding routing table in a second pingmessage and send the second ping message at some future point to thenode having ID 144. The node having ID 144 can receive the second pingmessage and can update its corresponding routing table with nodes in theliveness information included in second ping message (i.e., nodes in therouting table of the node having ID 174). The node having ID 144 canrepeat the sending of ping messages with other neighborhood nodes inring 1206.

It should be understood that when a newly joining node joins afederation, the newly joining node may not find an existing federationmember and thus becomes the sole member. Thus, there may be nopredecessor, successor, or neighbor nodes assigned for the newly joiningnode. Accordingly, the newly joining node is mapped as the best routingpartner in all cases.

Further, although joining a federation infrastructure has been describedwith respect to a single ring (ring 1106), it should be understood thatin some embodiments a node that joins one ring inherently also joins oneor more other rings. For example, referring briefly back to Figure '0, anode at joins ring '051 inherently also joins rings 1043, 1031, 1022,1011, and 1001. Thus, joining a federation infrastructure can beimplemented to join a plurality of rings. In other embodiments some orall of the acts of joining a federation infrastructure may be repeatedwhen joining multiple rings. For example, referring again to FIG. 10,joining a federation infrastructure can be repeated when a node joinsboth ring 1051 and ring 1014 (e.g., aliasing).

In any event, a joining node ID can be accessed and used to identify ajoining node in a sorted linked list as well as correspondinghierarchically partitioned sub-lists the joining node is to participatesin. A receiving node is identified from the sorted linked list and eachpartitioned sub-list. The join message is routed to a processing node(e.g., based on ID) in the sorted linked list and each portionedsub-list. A join response is received from the processing node in thesorted linked list and each partitioned sub-list.

Maintaining Membership In A Federation Infrastructure

Embodiments of the invention include a node maintaining membership in afederation infrastructure. Joining a federation infrastructure caninclude an act of issuing a join message to a federation infrastructure.A node maintaining membership in a federation infrastructure can includean act of sending a first ping message to a neighborhood node. The firstping message indicates that a current node sending the first pingmessage is neighbor of the neighborhood node. The first ping message canalso include routing partner and neighborhood nodes' state of thecurrent node. For example, in FIG. 11B, the node having ID 144 can senda ping message to the node having ID 151. Upon receiving the first pingmessage, the node having ID 151 is made aware that the node having ID144 is a neighbor of the node having ID 151. Node 151 may also discovernewer liveness information (for other nodes on ring 1206) from node 144as a side effect of this act.

Ping messages can be periodically repeated at a specified frequencybased on, for example, configuration state associated with a proximalring into which the ping message is to be sent. The frequency can bevaried depending on the configuration state. For example a specifiedping frequency for a WAN can be different than the specified frequencyfor a LAN. Ping messages can also be sent in accordance with a pingdistribution pattern. The ping distribution pattern for an originatingnode can indicate that ping messages are to be sent to be neighborhoodnodes in both directions on a ring. For example, the node having ID 144can send pings both in the direction of the node having ID 135 and inthe direction of the node having ID 151. Ping distribution patterns andfrequencies can be varied. For example, per proximity ring.

A node maintaining membership in a federation infrastructure can includean act of receiving a second ping message from the neighborhood node.The second ping message indicates to the current node at least that theneighborhood node originating the second ping message is a neighbor ofthe current node. The second ping message can also include routingpartner and neighborhood nodes' state of the originating neighborhoodnode. For example, the node having ID 151 can send a second ping messageto the node having ID 144. Upon receiving the second ping message, thenode having ID 144 is made aware that the node having ID 151 is aneighbor of the node having ID 144. The second ping message can alsoinclude liveness information for other nodes on ring 1106. Thusgenerally, ping messages can be exchanged within a neighborhood and canbe used to maintain neighborhood membership (for each proximalmembership) and an approximated common neighborhood view of nodepresence within the federation.

A received ping message can be periodically repeated/forwarded to othernodes within the proximal neighborhood into which the ping wasoriginated (sent by the originating node). Forwarded ping messages canalso be sent in accordance with a ping distribution pattern. The pingdistribution pattern for a forwarding node can indicate that pingmessages are to be sent to be neighborhood nodes in a direction awayfrom an originating node. For example, the node having ID 151 canforward pings originating at the node having ID 144 in the direction ofthe node having ID 174. Ping forwarding distribution patterns can bevaried, for example, per proximity ring.

Nodes can be configured to receive ping messages at correspondingintervals. When expected ping messages are not received, a node mayinterpret a communications failure and set the phase unknown indicationfor another node to true for the node that should have originated theexpected, but at least late, ping message.

A node maintaining membership in a federation infrastructure can includean act of proximally routing an update request message to a perfectrouting node. The update request message indicates to the routing nodereceiving such a routed update request that the current node isparticipating as a routing partner of the receiving routing node. Theupdate request message can also include at least the current node'sneighborhood nodes' identities (e.g. in the form of livenessinformation). For example, the node having ID 144 can route updatemessage 1116 to the node having ID 208 (the perfect routing partneroffset by 64 from 144). Because node 210 (a previously computed routingnode) is closest to 208, it will receive and process the routed updaterequest. Upon receiving update message 1116, the node having ID 210 ismade aware (or is reinforced) that the node having ID 144 is a routingpartner of the node having ID 210.

A node maintaining membership in a federation infrastructure can includean act of receiving an update response message from the processing(receiving) routing node. The update response indicates to the currentnode that the processing routing node is participating as a routingpartner of the current node. The update response message can alsoinclude at least the processing routing partner's neighborhood nodes'identifies. For example, the node having ID 210 can send update response1107 to the node having ID 144. Upon receiving update response 1107, thenode having ID 144 is made aware that the node having ID 210 is arouting partner of the node having ID 144.

A node maintaining membership in a federation infrastructure can alsoinclude an act of appropriately updating node information to indicatethat the current node and the neighborhood node are participating asneighbors and that the current node and the neighborhood node areparticipating as routing partners. For example, the node having ID 144can update node information corresponding to the node having ID 151 toindicate that the nodes having IDs 144 and 141 are participating in a(proximal) neighborhood. Similarly, the node having ID 144 can updatenode information corresponding to the node having ID 210 to indicatethat the nodes having IDs 144 and 210 are participating as routingpartners.

In some embodiments, application state saved at a specified node X isreplicated among its Neighborhood(X) nodes using reliable-floodingprotocol. Each item in the application state has an assigned owner,which could be the endpoint that created the item. Each item in theapplication state also has an associated timestamp (a.k.a. sequencenumber) given by its owner. The timestamp has at least three components:

-   -   Instance ID (e.g., an unsigned-integer) of the owning entity.        Must be at least monotonically (>1) increasing.    -   Sequence ID (e.g., a URI) identifying the particular sequence        generated by an owner. This component allows the same owner to        generate multiple independent sequences    -   Ordinal number (e.g., an unsigned-integer) identifying the        offset within the identified application sequence ID.

Item timestamps are used to detect latest information associated withthe corresponding item during replication because item timestampsgenerate at least a partial-order with <Instance ID, Sequence ID, andOffset> triples. The timestamp associated with an item being replicatedis compared against the local one, if any, to detect the latest one.Item timestamps are also used to support idempotent semantics ofcreate/update/delete operations. For example, when a node receives arequest to update an existing item in the application state, the updateis accepted only if the timestamp associated with the update request ishigher than the one associated with the local item. Conflict resolutiontechniques based on vector timestamps can be utilized where items cannotbe assigned a single owner. Application state replication providesfault-tolerance and facilitates load-balancing requests acrossneighborhood nodes.

As an optional behavior, Nodes not detecting (after a period of time) anexpected Update or Ping from (origin) other partner (routing and/orpartner) nodes can consider the phase-state unknown, set a phase unknownindication to true, and report it as such to other 3^(rd) party nodes.In other words periodic generation of updates and pings can be required.This requirement and actual timeout values can be an attribute ofvarious proximal rings. For example, a ring can have more restrictivetiming requirements for some sub-rings (e.g., in a LAN segment) and nodefailure detection/reporting is relatively quick. On the other hand, aring can have less restrictive timing requirements (or even no timingrequirements) for other sub-rings (e.g., on the Internet) and proactivenode failure detection/reporting is relative long (or doesn't exist).

Discovering Liveness In A Federation Infrastructure

Embodiments of the invention include a node discovering livenessinformation for another node in a federation infrastructure. Generally,referring to FIGS. 11A and 11B, any message, such as, for example, sync1103, sync response, 1104, update 1116, update response 1107, etc., caninclude at least one liveness header. In some embodiments, a livenessheader includes a <node ID, instance ID, phase [phase-statevalue].[phase unknown indication], freshness value, a color (proximity)value, and a weight value> for a node. In other embodiments, a livenessheader includes <a phase [phase-state value].[phase unknown indication],freshness value, a color (proximity) value, and a weight value>. Inthese other embodiments, liveness headers can be used to augmentaddressing headers that already include node ID and instance ID forsender and origin nodes. Since the addressing headers already includenode ID and instance ID, this information can be omitted from theliveness header.

A node discovering liveness information for another node can include anact of receiving a liveness header representing state information for anode participating in a federation infrastructure. The liveness headerincludes at a least a received participating node ID, a received node'sinstance ID, a received phase value, and a received freshness value. Forexample, the node having ID 144 can receive a first liveness header insync response 1104 from the node having ID 151. The first livenessheader can include a <participating node ID, an instance ID, phase value[phase-state value].[phase unknown indication], a freshness value, acolor (proximity) value, and a weight value> for the node having ID 174.The phase-state value (e.g., Inserting, Syncing, Routing, Operating)identifies the expressed phase of the node having ID 174 at the time ofthe first freshness value. The phase value (e.g., phase-state:[Inserting, Syncing, Routing, Operating], and phase unknown) identifiesthe expressed and/or detected phase information of the node having ID174 at the time indicated by the first freshness value.

However, a freshness value can be discounted due to communication delay.A freshness value can also decay with the passage of time. The decaycurves for a freshness value can differ (and may not be linear orsymmetric) for the different phase states (including unknown). Thus,across different node phases, the decay of a freshness value can benon-linear and/or asymmetric.

A node discovering liveness information for another node can include anact of accessing at least a current instance ID, current phase value,and current freshness value for the participating node maintained at thecurrent node. For example, the node having ID 144 can access a previousreceived and stored instance ID, phase value [phase-sate value.][phaseunknown indication], and freshness value for the node having ID 174.

A node discovering liveness information for another node can include anact of comparing at least the received instance ID, received phasevalue, and received freshness value to the current instance ID, thecurrent phase value, and the current freshness value respectively at acurrent node. For example, the node having ID 144 can compare thepreviously received and stored instance ID, phase value [phase-satevalue.][phase unknown indication], and freshness value for the nodehaving ID 174 to the instance ID, phase value [phase-sate value.][phaseunknown indication], and freshness value received in the livenessheader.

The node having ID 144 can determine that current state information forthe node having ID 174 (e.g., received from the node having ID 151) isstale based on (in order) the first instance ID being greater than thecurrently stored instance ID for the node having ID 174, based on firstphase-state value being more advanced than the currently storedphase-state value for the node having ID 174, or based on the firstfreshness value being a value greater than the freshness value currentlystored for the node having ID 174. The node having ID 144 can alsodetermine that at least one phase.unkown indication (either currentlystored or received in the liveness header) indicates that a phase-statewas known at the time the phase-state was detected/transmitted.

A node discovering liveness information for another node can include anact of determining if state information for the participating node is tobe updated at the current node based on the comparison. For example,based on the comparison of values for the node having ID 174, the nodehaving ID 144 can determine that state information for the node havingID 174 is to be updated. Updating outdated state information for thenode having ID 174 can include replacing current stored values (e.g.,for instance ID, phase-state value, phase unknown indication, orfreshness value) with values included in the liveness header. Forexample, the node having ID 144 can update state information for thenode having ID 174 to indicate that the node having ID 174 hastransitioned to a more advanced phase-state.

In some embodiments, it can be detected that communication with theparticipating node may have been lost. For example, the node having ID144 can detect that communication with the node having ID 151 has beenlost. Referring briefly to FIG. 13, in response to a prior subscriptionfor liveness events 1301 (with an endpoint of the node having ID 151),application layer 1752 can send endpoint down event 1303 (with anendpoint of the node having ID 151) to function layer 1351. In theseembodiments such detected liveness conditions can be indicated inliveness information with the Phase Unknown indicator being set to truealong with the last known Phase state value.

A node discovering liveness information for another node can furtherinclude an act of receiving a message that includes a second livenessheader from a second different node in the federation infrastructure.For example, the node having ID 144 can receive a status message (fromthe node having ID 103 or some other node of ring 1106) that includes asecond liveness header. The second liveness header can include <theparticipating node ID, a second instance ID, a second phase value[phase-state value].[phase unknown indication], a second freshnessvalue, a second color (proximity) value, and a second weight value> forthe node having ID 174. The second phase value (e.g., phase-state:[Inserting, Syncing, Routing, Operating], and phase unknown indication)identifies the expressed/detected phase of the node having ID 174 at thetime of the second freshness value.

Alternately, subsequent to receiving the first liveness header, the nodehaving ID 144 can attempt to communicate directly with the node havingID 174. If communication is successful, the node having ID 174 canreturn a message (e.g., sync response) having the node ID and secondinstance ID in an addressing header and having a liveness headerincluding <the second phase value, the second freshness value, thesecond color (proximity) value, and the second weight value>. If afailure is detected, the node having ID 144 generates an internalliveness state change (e.g. freshness=max, and phase unknownindication=true) and processes the state change as if the state changewere received from another node. Such a state change has highestfreshness value.

A node discovering liveness information for another node can alsoinclude an act of comparing the second instance ID, the second phasevalue, and the second freshness value to the current instance ID, thecurrent phase value, and the current freshness value respectively. Forexample, after receiving a status message from the node having ID 103,the node having ID 144 can determine that current state information forthe node having ID 151 is stale based on (in order) the second instanceID being greater than the first instance ID, the second phase being moreadvanced than the first phase value, or the second freshness value beinggreater than the first phase value.

A node discovering liveness information for another node can alsoincludes an act of determining if state information for theparticipating node is to be updated based on the comparison. Forexample, based on the comparison of values for the node having ID 174,the node having ID 144 can determine that state information for the nodehaving ID 174 is to be updated. Updating outdated state information forthe node having ID 174 can include replacing current stored values(e.g., for instance ID, phase-state value, phase unknown indication, orfreshness value) with values included in the second liveness header. Forexample, the node having ID 144 can update state information for thenode having ID 174 to indicate that the node having ID 174 hastransitioned to a more advanced phase-state.

In some embodiments, phase values are compared within the context ofequal color values. As previously described, a node can participate inmultiple proximity rings. Participation in multiple proximity rings canoccur as a result of participation in a more specific ring implyingparticipation in a more general ring (along a common spine). Forexample, referring back to FIG. 10, a node's participation in ring 532also implies that the node is participating in rings 1022, 1011, and1001. Thus, a color for a more specific ring also represents all parentproximal rings. Also as previously described, participation in multipleproximity rings can occur when a node in one ring is aliased into one ormore other rings (potentially along different spines). For example,still referring to FIG. 10, a node participating in ring 1032 can bealiased into ring 1031 (or even ring 1041 that would imply participationin rings 1031, 1022, 1011, and 1001). Thus, a color for one ring (e.g.,ring 1031) can be viewed as a peer color (or proximity) of another ring(e.g., ring 1032).

When a node participates in a plurality of proximity rings in an aliasedfashion, there is some potential that phase values (e.g., phase-statevalues and/or phase unknown indications) for the node will differbetween different proximity rings. Thus, a node that receives stateinformation for another node, identifies the corresponding proximityring for the state information (color) before determining if currentstate information is to be updated for that node and color. For example,the node having ID 144 can identify the corresponding proximity ring forreceived state information corresponding to the node having ID 174before comparing the received state information to current stateinformation.

Identifying an appropriate proximity ring can include comparing areceived color value to one or more current color values. When thereceived color value and a current color value are equal, other stateinformation, such as, for example, a current instance ID, a currentphase value, and a current freshness value, can be compared tocorresponding received state information, such as, for example, areceived instance ID, a received phase value, and a received freshnessvalue. On the other hand, when the received color value and a currentcolor value differ, further comparisons do not occur.

Equality between color values can result in a variety of ways. Forexample, equality between color values can result when a current colorvalue and a received color value indicate the same proximity ring (e.g.,ring 1032). Further, equality between color values can result when amore specific color value is compared to a corresponding parent colorvalue (e.g., another ring along the same spine). For example, comparingthe color value for ring 1032 to the color value for ring 1011 (or ring1022 or 1001) can result in equality. Thus, the child proximity is theparent proximity but is more specific.

Thus generally, currently operational nodes in a federationinfrastructure can exchange expressed and detected liveness stateinformation for other nodes even when communication with those othernodes appears to be lost.

Bootstrapping Mechanisms

Generally, in order for a node to become an active member of afederation (e.g., join), the node has to communicate with at least oneother node that is already an active member of the leaf ring it intendsto join. To help insure this initial form of communication is available,federations can utilize a bootstrapping mechanism. A bootstrappingmechanism can be used as a last resort when other types of communicationfail to identify an active member of a leaf ring or security constraintsrequire a newly joining node to initially communicate with at least oneof a set of special nodes such as seed nodes. That is when other typesof communication fail or because of security requirements, abootstrapping mechanism can be used to identify an active member node ofa leaf ring.

In some embodiments, seed nodes are used to bootstrap communication witha federation. Seed nodes provide well known points of entry for sometypes of cross (inter) proximity communication. Seed nodes help healring partitions due to infrastructure failure/recovery and generaldynamism. Each ring can have at least one operational seed node in orderto provide basic bootstrapping properties for a federation.

Peer seed nodes can communicate amongst themselves to maintain a ringstructure (e.g., a doubly linked list) for a proximity that consists ofat least all active seed nodes for that proximity. A dedicated seed nodesynchronization protocol can be used to provide each seed node with atleast total knowledge of all other seed nodes' presence (active) state.An active seed node is a member node of the proximity leaf ring in whichit is homed as well as all other ancestral rings of the leaf ring. Thus,a seed node can represent an entire spine of proximity rings, forexample, from the seed node's leaf ring to the root ring. Accordingly,seed nodes can function as highly available and well known entry nodesin each of those proximity rings. As a result, presence state about seednodes can be useful for various forms of communication (e.g.,inter-proximal communication) within a federation. Accordingly, seednodes can provide a number of special properties, such as, for example,acting as well known “join points” for joining nodes, acting as a securering authority, aiding in healing infrastructure partitions, and actingas a stable “entry node” for each of their proximities.

To provide presence data, a seed node's arrivals and orderly departurescan be registered as a stable entry node at a rendezvous point in eachof their proximities. For example, registration messages can be routedto a fixed URI whose destination ID is the SHA-1 hash of the string“Proximity:/”. While in one embodiment seed nodes acting as stable entrynodes register themselves in this manner there are other embodimentswhere selected non-seed nodes may also register themselves in the samemanner and with the same or similar protocols described here for seednode. When a stable entry node (such as a seed node) registers, thestable entry node can indicate each ring it is a member of Thus,information maintained at the rendezvous point identified by this fixedURI is essentially a list of stable entry nodes and their correspondingring memberships. Accordingly, any node can refer to the rendezvouspoint identified by this fixed URI to obtain a list of available stableentry nodes and their ring memberships.

In one embodiment the stable entry node directly registers these arrivaland departure events. In another embodiment, the stable entry noderegisters these events directly at a rendezvous point within itsimmediate proximity ring and that rendezvous point transparentlyfacilitates (directly or indirectly) updating of all other appropriaterendezvous points in each of the remaining proximities rings to whichthe registering/unregistering stable entry node belongs. The applicationstate sequencing and propagation properties of a federation can be usedto maintain and propagate this stable entry node registrationinformation. For example, a reliable-flooding protocol can be used toreplicate saved application state among a node's Neighborhood nodes.

The promotion of a stable entry node's presence data towards the rootring allows other nodes in a federation to look up at least one entrynode in every proximity. Entry Node Lookup can be facilitated by routinga node lookup message towards the above determined rendezvous point inthe Lowest Common Ancestor Ring (“LCAR”) of the leaf ring of the nodeperforming the lookup and the desired proximity ring. For example,referring to FIG. 10, a node in ring 1041 may desire to communicationwith a node in ring 1033. However, the node in ring 1041 may have nodirect knowledge of any node in ring 1033. Thus, the node in ring 1041can send a Node Lookup Message to ring 1022 (the LCAR of ring of ring1041 and ring 1033). A rendezvous point node in ring 1022 that processesentry node presence information (e.g. caused to exist in the systembecause of a registration message originated by that entry node) canreturn a Lookup Response Message with contact information for at least aregistered stable entry node in ring 1033.

In some embodiments, stable entry nodes are seed nodes configuredspecifically as stable entry nodes for maintaining presence data forvarious proximities. In other embodiments, other types of nodes can alsofunction as stable entry nodes maintaining presence data for variousproximities and may also be configured to perform other operations. Forexample, certain other types of nodes may be configured (e.g., by anadministrator) as being highly available and thus suitable as a stableentry node (i.e. to be registered as described above). However, theother types of nodes may not include additional seed node functionality(e.g., may not be trusted as a security ring authority). In someembodiments, rendezvous points that maintain entry node presence statefor their immediate proximity may register themselves as a stable entrynode in the ancestral ring or rings.

Node Monitoring

FIG. 15A illustrates an example ring architecture 1500 that facilitatesone node monitoring another node. As depicted, ring architectureincludes at least ring 1501 (and any number of other higher and/or lowerlevel rings (not shown)). Ring 1501 can be configured similar to ring806 of FIG. 8. However, monitoring can occur on any ring of nodes,including any of the rings in FIGS. 9, 10, 11A, and 11B. FIG. 15Adepicts an expanded view of subject node 1502 (having ID=83) and monitornode 1503 (having ID=2). In the depicted embodiment, monitor node 1503is to monitor subject node 1502. However, any node on ring 1501 can beconfigured to monitor any other node on ring 1501.

Embodiments of the invention include one node monitoring another node.One node monitoring another node can include an act of a subject nodegenerating a subject side time-to-live duration value for use inmonitoring of the subject node. For example, subject node 1502 canestablish time-to-live (TTL) duration value 1513. TTL duration value1513 indicates a duration for which subject node 1502 can assume amonitoring relationship with monitor node 1503 is active.

One node monitoring another node can include an act of the subject nodesending an establish request to the monitor node, the establish requestindicative of the subject node requesting that the monitor node monitorthe subject node, the establish request including the subject sidetime-to-live duration value. For example, subject node 1502 can sendestablish request 1522, including TTL value duration 1513, to monitornode 1503.

One node monitoring another node can include an act the subject nodeestablishing an existing subject side time-to-die time based on thesubject side time-to-live duration value and the time the establishrequest was sent, wherein the subject node clock reaching the existingsubject side time-to-die time, prior to receiving an establish grantfrom the monitor node, is an indication of the subject node having totransition to a failure state. For example, subject node 1502 canestablish subject side time-to-die time 1517 based on TTL duration value1513 and the time the establish request 1522 was sent to monitor node1503. Subject side time-to-die time 1517 can be a time relative tosubject node 1502. If a clock of subject node 1502 reaches subject sidetime-to-die time 1517, prior to receiving an establish grant formmonitor node 1503, subject node 1502 is to transition to a failurestate. In some embodiments, when a clock of subject node 1502 reachessubject side time-to-die time 1517, prior to receiving an establishgrant from monitor node 1503, a failure state is caused. In otherembodiments, other activities occur to transition subject node 1502 intoa failure state.

One node monitoring another node can include an act of the monitor nodereceiving the establish request from the subject node, the establishrequest indicative of the subject node requesting that the monitor nodemonitor the subject node, the establish request including at least thesubject side time-to-live duration value, the subject side time-to-liveduration value used to determine a subject side time-to-die time at thesubject node, wherein the subject node clock reaching the subject sidetime-to-die time, prior to receiving an establish grant from the monitornode, is an indication of the subject node having to transition to afailure state. For example, monitor node 1503 can receive establishrequest 1522, including TTL duration value 1513, from subject node 1502.TTL duration value 1513 having been used at subject node 1502 toestablish subject side time-to-die time 1517.

One node monitoring another node can include an act of the monitor nodederiving a monitor side time-to-live duration value from the subjectside time-to-live duration value. For example, monitor node 1502 can useTTL value duration 1513 to derive TTL duration value 1519. In someembodiments, monitor node 1503 copies TTL duration value 1513 to deriveTTL duration value 1519. In these embodiments, TTL value duration 1513and TTL value duration 1519 are equal. In other embodiments, monitornode 1503 modifies TTL duration value 1513 to derive TTL duration value1519. In these other embodiments TTL duration value 1513 and TTLduration value 1519 differ. For example, monitor node 1503 can increasethe value of TTL duration value 1513 to derive TTL duration value 1519such that TTL duration value 1519 is larger than TTL duration value1513.

One node monitoring another node can include an act of the monitor nodeestablishing a monitor side time-to-die time based on the monitor sidetime-to-live duration value and the time the establish request wasreceived, the monitor node clock reaching the monitor side time-to-dietime, prior to receiving a renew request from the subject node, beingindicative of a suspected failure of the subject node. For example,monitor node 1503 monitor side time-to-die time 1514 based on TTLduration value 1519 and the time establish request 1522 was received.Monitor side time-to-die time 1514 can be a time relative to monitornode 1503. If a clock of monitor node 1503 reaches monitor sidetime-to-die time 1514, prior to receiving a renew request from subjectnode 1502, monitor node 1503 suspects subject node 1502 of failure.

One node monitoring another node can include an act of the monitor nodesending an establish grant to the subject node to indicate to thesubject node that the monitor node has agreed to monitor the subjectnode. For example, monitor node 1503 can send establish grant 1523 tosubject node 1502. One node monitoring another node can include an actof the subject node receiving an establish grant from the monitor node,the establish grant indicative of the monitor node monitoring thesubject node. For example, subject node 1502 can receive establish grant1523 from monitor node 1503. Generally, establish grant 1523 indicatesthat monitor node 1503 has agreed to monitor subject node 1502. In someembodiments, the establish grant message can include the monitor sideTTL duration value. For example, it may be establish grant 1523 includesTTL duration value 1519.

Alternately, a monitor node can send an establish reject to a subjectnode to indicate to the subject node that the monitor node has notagreed to monitor the subject node. For example, in response toreceiving establish request 1522, monitor node 1503 can alternately (asindicated by the dashed line) send establish reject 1531 to subject node1502. A subject node can receive an establish reject sent from a monitornode. For example, subject node 1502 can receive establish reject 1531from monitor mode 1503. Establish reject 1531 generally indicates tosubject node 1502 that monitor node 1503 has not agreed to monitorsubject node 1502.

From time to time (and intermingled between the performance of otheroperations within ring architecture 1500), a subject node can renew anestablished monitoring agreement with a monitor node. Generally, thesubject node leaves the existing agreement in force (the currentsubject-side-time to die time) until a new grant is received. However,the subject node can generate a new TTL duration value and derive whatan updated time-to-die time would be. The subject node then sends thenew TTL duration value to the monitor node. The monitor node receivesthe new TTL duration value. When appropriate the monitor node grants therenew request and sends a renew grant back to the subject. The subjectnode receives the renew grant. In response to receiving the renew grantthe subject implements the renewed agreement using the updatedtime-to-die time as the new current time-to-die time.

As such, one node monitoring another node can include an act of thesubject node sending a renew request to the monitor node prior to thesubject node clock reaching the subject side time-to-die time. Forexample, subject node 1502 can send renew request 1515, including TTLduration value 1513, to monitor node 1503 prior to a clock of subjectnode 1502 reaching subject side time-to-die time 1517. In someembodiments, renew request 1515 does not include a subject side TTLduration value. In these embodiments, continued use of TTL durationvalue 1513 can be inferred. In other embodiments, TTL duration value1513 is expressly included in renew request 1515. In yet otherembodiments, a different subject side TTL duration value is included inrenew request 1515. A new subject side TTL duration value can begenerated and used by subject node 1502 in response to configurationchanges of subject node 1502 and/or to configuration changes else wherein ring 1501 (e.g., changed network conditions).

Node 1502 can also calculate what an updated subject side time-to-dietime is to be if a corresponding renew grant responsive to renew request1515 is received. The calculation can be based at least on the timerenew request 1515 was sent and on the subject side TTL duration valuerelated to or associated with renew request 1515.

One node monitoring another node can include an act of the monitor nodereceiving a renew request from the subject node subsequent to sendingthe establish grant message and prior to the monitor node clock reachingthe monitor side time-to-die time, the renew request indicating that thesubject node has not failed. For example, monitor node 1503 can receiverenew request 1515 subsequent to sending establish grant 1523 and priorto a clock of monitor node 1503 reaching monitor side time-to-die time1514. Reception of renew request 1515 can indicate to monitor node 1503that subject node 1502 has not failed.

One node monitoring another node can also include an act of the monitornode granting the renew request to the subject node. For example,monitor node 1503 can grant renew request 1515.

One node monitoring another node can include an act of the monitor nodeestablishing an updated monitor side time-to-die time in response to andbased at least on the time the renew request was received, the monitornode clock reaching the updated monitor side time-to-die time, prior toreceiving another renew request from the subject node, being indicativeof a suspected failure of the subject node. For example, monitor node1503 can establish updated monitor side time-to-die time 1521 inresponse to and based on the time renew request 1515 was received andthe implied or indicated monitor TTL duration value related to orpotentially contained in a renew request 1515. Updated monitor sidetime-to-die time 1521 can be a time relative to monitor node 1503.Updated monitor side time-to-die time 1521 can be subsequent to monitorside time-to-die time 1514. However, there is no requirement thatmonitor side time-to-die time 1514 have occurred before establishingupdated monitor side time-to-die time 1521. Thus, it is also possiblethat updated monitor side time-to-die time 1521 is in fact prior to (orthe same as) monitor side time-to-die time 1514. If a clock of monitornode 1503 reaches updated monitor side time-to-die time 1521, prior toreceiving another renew request from subject node 1502, monitor node1503 suspects subject node 1502 of failure.

If no subject side TTL duration value is included in renew request 1515(and thus TTL duration value 1513 is inferred) or if renew requestexpressly includes TTL duration value 1513, monitor node 1503 can alsouse TTL duration value 1519 to establish updated monitor sidetime-to-die time 1521. On the other hand, if a subject side TTL durationvalue other than TTL duration value 1513 is expressly included in renewrequest 1515, monitor node 1503 can use the other expressly includedsubject side TTL duration value to derive a new monitor side TTLduration value. From the new monitor side TTL duration value, monitornode 1503 can then establish updated monitor side time-to-die time 1521.

One node monitoring another node can include an act of the monitor nodesending a renew grant to the subject node to indicate to the subjectnode that the monitor node has agreed to continue monitoring the subjectnode. For example, monitor node 1503 can send renew grant 1527 tosubject node 1502. One node monitoring another node can include an actof the subject node receiving a renew grant from the monitor nodesubsequent to sending the corresponding renew request and prior to thesubject node clock reaching the subject side time-to-die time, the renewgrant message indicative of the monitor node continuing to monitor thesubject node. For example, subject node 1502 can receive renew grant1527 from monitor node 1503 subsequent to sending renew request 1515 andprior to a clock at subject node 1502 reaching subject side time-to-dietime 1517. Generally, renew grant 1527 is indicative of monitor node1503 agreeing to continue to monitor subject node 1502.

Alternately, a monitor node can send a renew reject to a subject node toindicate to the subject node that the monitor node is no longer agreeingto monitor the subject node. For example, in response to receiving renewrequest 1515, monitor node 1503 can alternately (as indicated by thedashed line) send renew reject 1533 to subject node 1502. A subject nodecan receive a renew reject sent from a monitor node. For example,subject node 1502 can receive renew reject 1533 from monitor mode 1503.Renew reject 1531 generally indicates to subject node 1502 that monitornode 1503 is no longer agreeing to monitor subject node 1502.

One node monitoring another node can include an act of the subject nodetransitioning to a previously calculated updated subject sidetime-to-die time in response to receiving the renew grant, wherein thesubject node clock reaching the updated subject side time-to-die time,prior to receiving another renew grant from the monitor node, is anindication of the subject node having to transition to a failure stateFor example, subject node 1502 can transition to updated subject sidetime-to-die time 1552 when the corresponding renew grant message isreceived. Updated subject side time-to-die time 1552 can have beencalculated at around the time renew request 1515 was sent to monitornode 1503. Updated subject side time-to-die time 1552 can have beencalculated based on the time corresponding renew request 1515 was sentand on the TTL duration related to or associated with renew request1515. Updated subject side time-to-die time 1552 can be a time (e.g.,subsequent, prior, or equal to subject side time-to-die time 1517)relative to subject node 1502.

If TTL value 1513 is still the appropriate TTL value, subject node 9102can also used TTL duration value 1513 to establish updated subject sidetime-to-die time 1552. If another TTL duration value has been generated,subject node 1502 can also use the other generated TTL duration value toestablish updated subject side time-to-die time 1552.

Subsequent, to establishment of a current subject side time-to-die time(either 1517 or 1552), it may be a clock at subject node 1502 reachesthe current subject side time-to-die time prior to receiving anotherrenew grant from monitor node 1503. This may result from communicationerrors between subject node 1502 and monitor node 1503. For example,subject node 1502 may send another renew request subsequent receivingrenew grant 1527 and prior to a clock of subject node 1502 reachingupdated subject side time-to-die time 1552. However, due tocommunication failures the other renew request does not reach monitornode 1503. Alternately, the other renew request may be received atmonitor node 1503, but the corresponding renew grant from monitor node1503 does not reach subject node 1502 due to communication errors. Ineither event, a clock at subject node 1502 may reach a current subjectside time-to-die time prior to receiving the corresponding renew grantresponsive to the other renew request.

Alternately, subject node 1502 can malfunctioning such that subject node1502 is prevented from sending another renew request to monitor node1503 prior to a clock at subject node 1502 reaching t updated subjectside time-to-die time 1552.

However, whether or not a renew request is sent, if a renew grant is notreceived prior to a clock at subject node 1502 reaching a currentsubject side time-to-die time 1552, subject node 1502 transitions into afailure state.

Referring back to monitor node 1503, it may be that a clock at monitornode 1503 reaches a monitor side time-to-die time (either 1514 or 1521)prior to receiving another renew request from subject node 1502 (eitherdue to a malfunction at subject node 1502 or to communication errors inring 1501). As a result, monitor node 1503 suspects subject node 1502 offailure. Monitoring node 1503 can transition to a timeout stateindicative of detecting a suspected failure at another node.

Two-Way Node Monitoring

In other embodiments a pair of nodes can monitor each other. Thus, afirst node can monitor a second node and the second node can alsomonitor the first node. For example, each node can implement both thesubject node side and the monitor node side of nod monitoring with theother node. FIG. 15B illustrates an example ring architecture 1500 thatfacilitates two nodes monitoring each other.

Node 1571 can generate TTL duration value 1529 for use in monitoringnode 1571. Node 1571 can send establish request 1562, including TTLduration value 1529, to node 1572. Node 1571 can also establish subjectside time-to-die time 1573 based on TTL duration value 1529. Node 1572can receive establish request 1562, including TTL duration value 1529,from node 1571. Node 1572 can derive TLL duration value 1549 from TTLduration value 1529. Node 1572 can establish monitor side time-to-dietime 1539 based on TTL duration value 1549. Node 1572 can send establishgrant 1574 to node 1571. Node 1571 can receive establish grant 1574 fromnode 1572.

In parallel, node 1572 can generate TTL duration value 1575 for use inmonitoring node 1572. Node 1572 can send establish request 1526,including TTL duration value 1575, to node 1571. Node 1572 can alsoestablish subject side time-to-die time 1535 based on TTL duration value1575. Node 1571 can receive establish request 1526, including TTLduration value 1575, from node 1572. Node 1571 can derive TLL durationvalue 1553 from TTL duration value 1533. Node 1571 can establish monitorside time-to-die time 1537 based on TTL duration value 1553. Node 1571can send grant message 1576 to node 1572. Node 1572 can receive grantmessage 1576 from node 1571.

Alternately, either of nodes 1571 and 1572 reject an establish requestfrom the other node. For example, node 1571 can reject establish request1562. Likewise, node 1572 can reject establish request 1526. When eithernode rejects an establish request, it can send an establish reject(e.g., similar to establish reject 1531) to the other node. Thisindicates to the other node that no monitoring agreement has beenestablished.

Node 1571 and node 1572 can then exchange renew requests and renewgrants (as well as renew rejects similar to renew reject 1533) aspreviously described. Accordingly, each of node 1571 and node 1572 areboth a subject node and a monitor node. Based on the depicted TTLduration values and time-to-die times in FIG. 15B, various events mayoccur during and/or after the monitor relationships are established.

If a clock at node 1571 reaches subject side time-to-die time 1573 priorto receiving a renew grant from node 1572, node 1571 transitions to afailure state. If a clock at node 1572 reaches monitor side time-to-dietime 1539 prior to receiving a renew request from node 1571, node 1572suspects node 1571 of failure.

If a clock at node 1572 reaches subject side time-to-die time 1535 priorto receiving a renew grant from node 1571, node 1572 transitions to afailure state. If a clock at node 1571 reaches monitor side time-to-dietime 1537 prior to receiving a renew request from node 1572, node 1571suspects node 1572 of failure.

Arbitration of Node Failures

Due to various different types of communication errors and nodemalfunctions, there exists some possibility that each node in a pair ofnodes will suspect failure of the other node. Further, each node maysuspect that it is functioning properly.

In some ring architectures, portions of resources are configured suchthat a single node controls a resource at a given moment in time.Further, the needed availability of some resources may also be high suchthat essentially constant control by a node is required. Thus, when anode fails, control of various resources may need to be transferred toanother node. Accordingly, when a node in a pair of nodes suspects theother node of failure, arbitration mechanisms can be used to determineat least which node has or should fail.

For example, when each node when a pair nodes suspects the other node offailing, each node can transition to a timeout state and report theirsuspicion to an arbitration facility. When in a timeout state, certainother processing at each node can be suspended until the results of thearbitration are received. The arbitration facility can report back to anode indicating if it is to remain active. For example, an arbitrationfacility can send an accept message to a reporting node that is toremain active. The arbitration facility can send a deny message to areporting node that is to transition to a failure state. A node thatreceives an accept message can remain active. A node that doesn't notreceive an accept message (e.g., due to network conditions) or thatreceives a deny message transitions to a failure state.

FIG. 15C illustrates example ring architecture 1500 that facilitatesarbitration when mutually monitoring nodes each can report that theother node is suspected of failing. FIG. 15C depicts an expanded view ofnode 1581 (having ID=98), monitor node 1582 (having ID=64), andarbitrator 1983.

In some embodiments, arbitrator 1583 is also a member of ring 1501. Inother embodiments, arbitrator 1583 is a member of an ancestor ring ofring 1501 but is not member of ring 1501. In further embodiments,arbitrator 1583 is external to the ring hierarchy that includes ring1501. For example, arbitrator 1583 can be included in a separatearbitration federation ring of nodes. Nodes in the arbitrationfederation can be configured as arbitrators for the nodes of ring 1501and its ancestors.

In some embodiments, arbitrator 1583 is mutually agreed to by node 1571and node 1582 to arbitrate for nodes 1581 and 1582. In otherembodiments, arbitrator 1583 is assigned to arbitrate for nodes 1581 and1582 by another entity. The other entity can be a node internal to thering hierarchy including ring 1501 (e.g., a seed node) or, for example,a human administrator. For example, the other node can be a member ofring 1501 or a member of an ancestor ring of ring 1501 but not member ofring 1501. Alternately, the other entity can be external the ringhierarchy including ring 1501. For example, the other entity can be anode that is a member of separate arbitration federation ring.

Arbitrator 1583 can have varying knowledge of the ring hierarchyincluding ring 1501. For example, arbitrator 1583 can have globalknowledge of the ring hierarchy including ring 1501. Alternately,arbitrator 1583 can have knowledge of some subset of rings included thering hierarchy including ring 1501. In other embodiments, arbitrator1583 has knowledge of a subset of nodes in ring 1501 including (andpotentially only) nodes 1581 and 1582.

Arbitrator 1583 can be configured to arbitrate for any number of nodepairs including, but not limited to, nodes 1581 and 1582. In someembodiments, an arbitration mechanism has no knowledge of nodes it is toarbitrate for prior to receiving a report of a suspected node failure.Thus, although a pair of nodes have agreed to use arbitrator 1583 orarbitrator 1583 has been assigned to arbitrate for a pair of nodes,arbitrator 1583 may be unaware of any agreement or assignment prior toreceiving a report of a suspected node failure for a node in the pair ofnodes.

Arbitration can include arbitrating between nodes that presentconflicting failure reports. For example, when a first node ismonitoring a second node and the second node is also monitoring thefirst node, it may be that each node reports that the other node issuspected of failure. The suspected failure can be detected usingvirtually any failure detection mechanisms including those previouslydescribed in this document.

Failed node list 1547 can include a list of nodes that have beenreported as suspected failed nodes. Nodes can be report other nodes assuspected failed nodes to arbitrator 1583 and, when appropriate,arbitrator 1583 can include the reported nodes in failed node list 1547.Arbitrator 1583 can remove failed nodes from failed node list 1547 afterappropriate periods of time (e.g., at a future time when the likelihoodof continued conflict is not possible). For example, entries in failednode list 1547 can be removed at recovery time interval 1542 after theywere inserted into failed node list 1547. Recovery time interval 1542can be long enough to insure that nodes that have been told to fail dofail.

Thus, embodiments of the invention include arbitrating betweenconflicting reports of suspected node failures. Arbitrating betweenconflicting reports of suspected node failures can include an act of afirst node sending a report to an arbitration facility that a secondnode is suspected of failing. For example, node 1581 can send report1534 to arbitrator 1583. Arbitrating between conflicting reports ofsuspected node failures can include an act of an arbitrator receiving areport from the first node that the second node is suspected of failing.For example, arbitrator 1583 can receive report 1534 from node 1581.

Arbitrating between conflicting reports of suspected node failures caninclude an act of the arbitrator determining that no other node hassuspected the first node of failing within a specified recovery timeinterval prior to receiving the report from the first node. For example,arbitrator 1583 can determine that no other node has suspected node 1581of failing within recovery time interval 1542 (after which arbitrator1583 would have removed node 1581 from failed node list 1547 anyway).

Arbitrating between conflicting reports of suspected node failures caninclude an act of the arbitrator recording in a list that the secondnode is in a failure state. For example, arbitrator 1583 can record infailed node list 1547 that node 1582 (ID=64) is in a failure state.

Arbitrating between conflicting reports of suspected node failures caninclude an act of the arbitrator sending an accept message to the firstnode within a maximum response time interval, the accept messageincluding a failure time value indicative of a time period after whichthe second node is guaranteed to transition into a failure state. Forexample, arbitrator 1583 can send accept message 1584 to node 1581within maximum response time interval 1543 of receiving report 1534.Accept message 1584 includes failure time interval 1536 indicative of atime when node 1582 is guaranteed to have transitioned into a failurestate. Generally, a maximum response time interval represents a point intime after which a requestor (e.g., node 1581 or 1582) assumes anarbitration facility (arbitrator 1583) will not answer a request forarbitration (e.g., report 1534 or 1538). When a maximum response timeinterval expires at a requestor subsequent to sending a request forarbitration, the requestor performs similar (and potentially identical)operations to those that would be performed if an express deny messagewas received.

Arbitrating between conflicting reports of suspected node failures caninclude an act of the first node receiving an accept message from thearbitration facility within a maximum response time interval, the acceptmessage including a time value indicative of a time period after whichthe second node is guaranteed to transition into a failure state. Forexample, node 1581 can receive accept message 1584, including failuretime interval 1536, from arbitrator 1583. Failure time interval 1536 isindicative of a time when node 1582 is guaranteed to have transitionedinto a failure state. Thus, after the expiration of failure timeinterval 1536, node 1581 can attempt to claim control of one or morering resources previously controlled by node 1582.

Arbitrating between conflicting reports of suspected node failures caninclude includes an act of the first node claiming control of one ormore ring resources previously controlled by the second node subsequentto expiration of the time period. For example, node 1581 can claimcontrol of one or more ring resources within ring 1501 previouslycontrolled by the node 1582 subsequent to expiration of failure timeinterval 1536.

Claimed ring resources can vary depending on the ring resourcescontrolled by node 1582 prior to transition to a failure state. Forexample, node 1581 can assume message routing responsibilities of node1582 (e.g., the responsibility to receive messages directed to a rangeof identifies on ring 1501), any seed node responsibilities of node1582, any arbitration responsibilities of node 1582, etc.

At some time at or after the first node reports the second node, thesecond node may also suspect the first node of failure. For example, itmay be that node 1582 also suspects node 1581 of failure.

Arbitrating between conflicting reports of suspected node failures caninclude an act of the second node sending a report to the arbitrationfacility that the first node is suspected of failing. For example, node1582 can send report 1538 to arbitrator 1583 that node 1581 is suspectedof failure. Arbitrating between conflicting reports of suspected nodefailures can include an act the arbitrator receiving a report from thesecond node that the first node is suspected of failing, the report fromthe second node received within the specified recovery time intervalsubsequent to receiving the report from the first node. For example,arbitrator 1583 can receive report 1538 from node 1582 that node 1581 issuspected of failure within recovery time interval 1542 of receivingreport 1534.

Arbitrating between conflicting reports of suspected node failures caninclude an act of the arbitrator referring to the list to determine thatthe second node is to transition to a failure state. For example,arbitrator 1583 can refer to failed node list 1547 to determine thatnode 1582 (ID=64) is to transition to a failure state.

Arbitrating between conflicting reports of suspected node failures caninclude an act of sending a deny message to the second node to cause thesecond node to transition into a failure state. For example, arbitrator1583 can send deny message 1585 to node 1582 to cause node 1582 totransition to a failure state. Arbitrating between conflicting reportsof suspected node failures can include an act of the second nodereceiving a deny message from the arbitration facility. For example,node 1582 can receive deny message 1585 from arbitrator 1583.

Arbitrating between conflicting reports of suspected node failures caninclude an act of the second node transitioning into a failure state.For example, node 1582 can transition into a failure state in responseto receiving deny message 1585. After failing, node 1582 cansubsequently attempt to rejoin ring 1501.

Routing In Accordance With Cached Agreements

In some embodiments, messages are routed in accordance with cachedrouting agreements. For example, adjacent nodes of a ring can agree to adivision of responsibility for a range of unoccupied identifiers betweenthe adjacent nodes. An identifier can be unoccupied for any number ofreasons. For example, an identifier may be unoccupied because theidentifier is unassigned (i.e., the identifier that has not beenassigned to a node). For assigned identifiers (i.e., identifiers thathave been assigned to a node), an identifier may be unoccupied becausethe corresponding node has been deliberately shutdown or the node is forsome reason, such as, for example, due to communication or nodefailures, otherwise unreachable.

Routing agreements between nodes can be established and cached prior tonodes being permitted to accept messages for and deliver messages forany of the unoccupied identifiers that are to be the responsibility ofthe adjacent nodes. Reference to a cached routing agreementsignificantly reduces any communication between (potentially) adjacentnodes that may otherwise occur to determine which node is responsiblefor a specific unoccupied identifier.

A cached routing agreement can divide a range of unoccupied identifiersin an arbitrary fashion, in accordance with configurable rules, or inaccordance with a fixed methodology. In some embodiments, a range ofidentifiers between adjacent nodes on a ring is divided essentially inhalf. This reduces the likelihood of an unoccupied identifier beingfurther from a node that is responsible for the unoccupied identifier.

When there is an even number of unoccupied identifiers between adjacentnodes, the midway point between the adjacent nodes is between unoccupiedidentifiers. Thus, responsibility for the unoccupied identifiers can bedivided at the midway point between the adjacent nodes. Accordingly,each adjacent node can be assigned responsibility for an equal number ofunoccupied identifiers.

On the other hand, when there is an odd number of unoccupied identifiersbetween adjacent nodes, the midway point between the adjacent nodes isat an unoccupied identifier. Thus, responsibility for the unoccupiedidentifiers can be divided at one side or the other of the unoccupiedidentifier that is the midway point. Accordingly, one adjacent node canbe assigned responsibility for one more unoccupied identifier than theother adjacent node.

For example, referring now to FIG. 16A, FIG. 16A illustrates an examplering architecture 1600 that facilitates routing a message in accordancewith a cached two-way agreement between nodes. As depicted, variousnodes (shown as squares on ring 1650) including (but not limited to)nodes 1601, 1602, 1603, 1661, 1662, and 1663 are included on ring 1650.Each node has a corresponding ID (shown in parenthesis) indicating itsposition on ring 1650. For example, node 1601 has ID=64 and node 1602has ID=30.

There are ranges of unoccupied identifiers between the depicted nodes.For example, unoccupied identifier range 1611 represents unoccupiedidentifiers 31 through 63 between nodes 1602 and 1601.

As depicted, node 1601 and 1602 have established and cached two-wayagreement 1623. For example, through prior communication, nodes 1601 and1602 can determine that there are no other nodes currently interspersedbetween ID=64 and ID=30. Thus, nodes 1601 and 1602 can further determinethat they are adjacent to one another on ring 1650. Accordingly, node1601 and 1602 can divide responsibility for unoccupied identifier range1611 (i.e., unoccupied identifiers 31 through 63) such that node 1602 isresponsible for a portion of unoccupied identifier range 1611 and node1601 is responsible for the remaining portion unoccupied identifierrange 1611. Each node is also responsible for its assigned ID. That is,node 1602 is responsible for ID=30 and node 1601 is responsible forID=64.

Accordingly, as depicted by responsibility boundary 1613 (betweenunoccupied identifier 47 and unoccupied identifier 48), node 1602(ID=30) is responsible for itself as well as unoccupied identifiers 31through 47 and node 1601 (ID=64) is responsible for itself as well asunoccupied identifiers 48 through 63. Although the midway point betweennodes 1601 and 1602 is at unoccupied identifier 47, node 1602 isassigned responsibility for unoccupied identifier 47 such that eachunoccupied identifier is the responsibility of a single node. Thus, aspreviously described, when a responsibility boundary falls on anunoccupied identifier, one of the adjacent nodes can be assign the soleresponsibility for the unoccupied identifier.

Embodiments of the invention include routing a message in accordancewith a cached two-way agreement. Routing a message in accordance with acached two-way agreement can include an act of a receiving nodereceiving a message along with a destination identifier indicating adestination on the ring of nodes, the destination identifier locatedbetween the receiving node and one of the immediate neighbor nodes. Forexample, node 1601 can receive message 1651, indicated for delivery toID=55. Alternately, node 1601 can receive message 1652, indicated fordelivery to ID=39. Message 1651 and 1652 can be received from anothernode in ring 1650 (intra-ring communication), from a node in anotherring of ring architecture 1600 (inter-ring communication), or throughnon-ring communication.

Routing a message in accordance with a cached two-way agreement caninclude an act of the receiving node referring to a cached two-wayagreement between the receiving node and the immediate neighbor node todetermine the next appropriate node that is to receive the message. Thetwo-way agreement at least implies a division of responsibility for theidentifier space between the receiving node and an immediate neighbornode. For example, node 1601 can refer to cached two-way agreement 1623to determine the next appropriate node that is to process message 1651.Since cached two-way agreement 1623 indicates that node 1601 (ID=64) isresponsible for unoccupied identifier 55, node 1601 determines that itis the appropriate node to process message 1651. Likewise, node 1601 canrefer to cached two-way agreement 1623 to determine the next appropriatenode that is to process message 1652. Since cached two-way agreement1623 indicates that node 1602 (ID=30) is responsible for unoccupiedidentifier 39, node 1601 determines that node 1602 is the nextappropriate node that is to process message 1652.

Routing a message in accordance with a cached two-way agreement caninclude an act of sending the message to the next appropriate componentbased on the determination of the next appropriate node. For example,node 1601 can provide message 1651 to its resource handler instancecorresponding to unoccupied identifier 55, since cached two-wayagreement 1623 indicates that node 1601 is responsible for unoccupiedidentifier 55. Alternately, node 1601 can provide message 1652 to node1602, since cached two-way agreement 1623 indicates that node 1602 isresponsible for unoccupied identifier 39. Subsequently, node 1602 canprovide message 1652 to its resource handler instance corresponding tounoccupied identifier 39.

When an identifier is not included in a cached two-way agreement, a nodecan refer to a routing table (e.g., as depicted in FIG. 8) to makeprogress towards a destination. For example, node 1601 can send message1653, indicated for delivery to ID=203, to node 1661 (ID=200). Node 1661can then refer to any cached two-way agreements with its adjacent nodesto determine the node that is responsible for identifier 203.

In some embodiments, multiple two-way agreements can, from theperspective of a given node, essentially represent a three-way agreementbetween the given node, the given node's immediate predecessor node, andthe given node's immediate successor node. FIG. 16B illustrates theexample ring architecture 1600 that facilitates routing a message inaccordance with multiple cached two-way agreements.

As previously described, nodes 1601 and 1602 can establish cachedtwo-way agreement 1623. Similarly, nodes 1601 and 1603 can establishcached-two way agreement 1624 to divide responsibility for unoccupiedidentifier range 1612 (i.e., unoccupied identifiers 65 through 101).Thus, through prior communication, nodes 1601 and 1603 can determinethat there are no other nodes currently interspersed between ID=65 andID=101. Thus, nodes 1601 and 1603 can further determine that they areadjacent to one another on ring 1650. Accordingly, nodes 1601 and 1603can divide unoccupied identifier range 1612 such that node 1602 isresponsible for a portion of unoccupied identifier range 1612 and node1601 is responsible for the remaining portion of unoccupied identifierrange 1612. Accordingly, as depicted within two-way agreement 1624, node1601 (ID=64) is responsible for itself as well as unoccupied identifiers65 through 82 and node 1602 (ID=101) is responsible for itself as wellas unoccupied identifiers range 83 through 100.

From the perspective of node 1601, the combination of cached two-wayagreement 1623 and cached two-way agreement 1624 essentially representsthree-way agreement 1673. That is, node 1601 is responsible for aportion of identifier space between node 1601 and node 1602 and isresponsible for a portion of identifier space between node 1601 and node1603. The parenthetical ranges of identifiers indicate the ranges ofresponsibility (i.e., 47 through 64 and 64 through 82) form thecached-two way agreements 1623 and 1624 on either side of node 1601.

Embodiments of the invention include routing a message in accordancewith a multiple cached two-way agreements.

Routing a message in accordance with a multiple cached two-wayagreements can include an act of a receiving node receiving a messagealong with a destination identifier indicating a destination on the ringof nodes. For example, node 1601 can receive any of messages 1651, 1652,1653, 1654, and 1656 indicated for delivery to ID=55, ID=39, ID=203,ID=74, and ID=94 respectively. Messages 1651, 1652, 1653, 1654, and 1656can be received from another node in ring 1650 (intra-ringcommunication) or from a node in another ring of ring architecture 1600(inter-ring communication), or through non-ring communication.

Routing a message in accordance with a multiple cached two-wayagreements can include an act of the receiving node referring to a firstcached two-way agreement with the predecessor node and a second cachedtwo-way agreement with the successor node to determine the nextappropriate node that is to receive the message. The first and secondcached two-way agreements at least imply a division of responsibilityfor the identifier space between the predecessor node and the successornode. For example, node 1601 can refer to cached three-way agreements1623 and 1624 to determine the next appropriate node that is to receiveany of messages 1651, 1652, 1653, 1654, and 1656.

Since cached two-way agreement 1623 indicates that node 1602 (ID=30) isresponsible for unoccupied identifier 39, node 1601 determines that node1602 is the next appropriate node that is to process message 1652. Sincecached two-way agreement 1623 indicates that node 1601 (ID=64) isresponsible for unoccupied identifier 55, node 1601 determines that itis the appropriate node to process message 1652. Since cached two-wayagreement 1624 indicates that node 1601 (ID=64) is responsible forunoccupied identifier 74, node 1601 determines that it is theappropriate node to process message 1654. Since cached two-way agreement1624 indicates that node 1603 (ID=101) is responsible for unoccupiedidentifier 94, node 1601 determines that node 1603 is the nextappropriate node that is to process message 1654.

Routing a message in accordance with a multiple cached two-wayagreements can include an act of sending the message to the nextappropriate component based on the determination of the next appropriatenode. For example, node 1601 can send messages 1651, 1652, 1653, 1654,and 1656 to the next appropriate component on ring 1650 based on thedetermination of the next appropriate node that is to process messages1651, 1652, 1653, 1654, and 1656.

For example, node 1601 can provide message 1652 to node 1602, sincecached two-way agreement 1623 indicates that node 1602 is responsiblefor unoccupied identifier 39. Subsequently, node 1602 can providemessage 1652 to its resource handler instance corresponding tounoccupied identifier 39. Node 1601 can provide message 1651 to itsresource handler instance corresponding to unoccupied identifier 55,since cached two-way agreement 1623 indicates that node 1601 isresponsible for unoccupied identifier 55. Node 1601 can provide message1654 to its resource handler instance corresponding to unoccupiedidentifier 74, since cached two-way agreement 1624 indicates that node1601 is responsible for unoccupied identifier 74. Node 1601 can providemessage 1656 to node 1603, since cached two-way agreement 1624 indicatesthat node 1603 is responsible for unoccupied identifier 94.Subsequently, node 1603 can provide message 1656 to its resource handlerinstance corresponding to unoccupied identifier 94.

When an identifier is not included in a cached either of multiple cachedtwo-way agreements, a node can refer to a routing table (e.g., asdepicted in FIG. 3) to make progress towards a destination. For example,node 1601 can send message 1656, indicated for delivery to ID=203, tonode 1661 (ID=200). Node 1661 can then refer to a any cached two-wayagreements with its predecessor node and/or its successor node todetermine the next appropriate component that is to receive message1653.

Formulating Cached Agreements

Rings can be reconfigured from time to time, such as, for example, whena new node joins a ring or when an existing node departs a ring (e.g.,through graceful removal, as a result of node monitoring, throughreference to an arbitrator, etc.). When a node detects that theconfiguration of a ring has changed, the node can reformulate cachedrouting agreements with any adjacent nodes. During agreementreformulation, the node can queue any received messages, expect thosefor formulating the agreement. After formulation of the agreement iscomplete, the node can then process the messages in accordance with theagreement.

Reconfiguration of a ring can cause multiple routing agreements to bereformulated. For example, when a node departs a ring, immediatelyadjacent nodes on either side of the departing node can formulate anagreement for the range of unoccupied identifiers that were previouslythe responsibility of the departing node (thus potentially gainingresponsibility for additional unoccupied identifiers). Thisreformulation joins responsibility for a portion of the range ofunoccupied identifiers from the departing node with the range ofunoccupied identifiers for each immediately adjacent node. That is, eachimmediately adjacent node gains responsibility for a portion of thedeparting node's range of unoccupied identifiers and the departingnode's identifier.

FIGS. 17A through 17D illustrate an example ring architecture 1700 thatfacilitates formulating a cached two-way agreement. As depicted in FIG.17A, nodes 1701 and 1702 have formulated cached two-way agreement 1723dividing responsibility for unoccupied identifier range 1712 (i.e.,unoccupied identifiers 31 through 63) at responsibility boundary 1713(between unoccupied identifier 47 and unoccupied identifier 48).Similarly, nodes 1702 and 1762 have formulated cached two-way agreement1743 dividing responsibility for unoccupied identifier range 1711 (i.e.,unoccupied identifiers 255 through 29) at responsibility boundary 1733(between unoccupied identifiers 14 and 15).

At some time subsequent to the formulation of cached two-way agreements1723 and 1743, node 1702 can leave ring 1750 (e.g., through gracefulremoval, as a result of node monitoring, based on instructions from anarbitrator, etc.). Referring now to FIG. 17B, subsequent to node 1702leaving ring 1750 there is no node responsible for the unoccupiedidentifiers that were previously the responsibility of node 1702.Unoccupied identifier range 1713 (unoccupied identifiers 15 through 47,including now unoccupied identifier 30) represents the range ofunoccupied identifiers that node 1702 was responsible for prior todeparting ring 1750.

In response to node 1702 leaving ring 1750, nodes 1701 and 1762 attemptto identify new immediate neighbor nodes. Node 1762 attempts to identifya new immediate successor node (i.e., an immediate neighbor node in thesame direction as node 1702 relative to node 1762). Node 1701 attemptsto identify a new immediate predecessor node (i.e., an immediateneighbor in the same direction as node 1702 relative to node 1701). InFIG. 17B, node 1762 identifies node 1701 as its new immediate successorand node 1701 identifies node 1762 as its new immediate predecessor.

Upon identifying new immediate neighbor nodes, nodes 1762 and 1701formulate cached two-way agreement 1763 to that divides responsibilityfor unoccupied identifier range 1714 (unoccupied identifiers 255 through63, including now unoccupied identifier 30). Unoccupied identified range1714 includes unoccupied identifier range 1713, which was previously theresponsibility of node 1702. Thus, portions of unoccupied identifierrange 1713 can become the responsibility of either node 1762 or node1701, after node 1702 departs ring 1750.

Accordingly, as depicted by responsibility boundary 1753 (betweenunoccupied identifier 31 and unoccupied identifier 32), node 1762(ID=254) and node 1701 (ID=30) formulate cached two-way agreement 1763.In accordance with cached two-way agreement 1763, node 1762 (ID=254) isresponsible for itself as well as unoccupied identifiers 255 through 31and node 1701 (ID=64) is responsible for itself as well as identifierrange 32 through 63. Although the midway point between nodes 2201 and2202 is at unoccupied identifier 31, node 1762 is assignedresponsibility for unoccupied identifier 31 such that each unoccupiedidentifier is the responsibility of a single node.

During time between the departure of node 1702 and formulation of cachedtwo-way agreement 1763, nodes 1701 and 1762 do not process messagesindicated for delivery to identifiers in the range between 255 and 63.Instead, nodes 1701 and 1762 queue any messages, expect those forformulating cached two-way agreement 1763. After formulation of thecached two-way agreement 1763 is complete, nodes 1701 and 1762 can thenprocess the messages in accordance with cached two-way agreement 1763.

When a new node joins a ring between two existing nodes, each existingnode can formulate a routing agreement with the new node (and thuspotentially giving up responsibility for a portion of unoccupiedidentifiers). This formulation can essentially split a range ofunoccupied identifiers an existing node is responsible for between thejoining node and the existing node. That is, each existing nodepotentially gives up responsibility for a portion of the existing node'sunoccupied identifiers to the joining node.

Referring now to FIG. 17C, at some time subsequent to the formulation ofcached two-way agreement 1763, node 1704 (ID=44) can join ring 1750.Subsequent to node 1704 joining ring 1750, node 1762 can detect node1704 as its immediate successor. Likewise, node 1701 can detect node1704 as its immediate predecessor. In response to each of thedetections, unoccupied identifier range 1714 is essentially split intounoccupied identifier range 1715 (unoccupied identifiers 255 through 43)and unoccupied identifier range 1716 (unoccupied identifiers 45 through63). New cached-two way agreements can then be formulated to divideresponsibility for unoccupied identifier ranges 1715 and 1716.

Referring now to FIG. 17D, upon identifying node 1704 as a new immediatesuccessor node, nodes 1762 and 1704 formulate cached two-way agreement1794 to that divides responsibility for unoccupied identifier range 1715(unoccupied identifiers 255 through 43). Unoccupied identified range1715 includes portions of unoccupied identifier range 1714, which werepreviously the responsibility of node 1762 and in this case some ofwhich were previously the responsibility of node 1701. Thus, portions ofunoccupied identifier range 1714 that were the responsibility of eithernode 1762 or node 1701, can become the responsibility of node 1704 whennode 1704 joins ring 1750.

Accordingly, as depicted by responsibility boundary 1793 (betweenunoccupied identifier 17 and unoccupied identifier 24), node 1762(ID=254) and node 1704 (ID=44) formulate cached two-way agreement 1794.In accordance with cached two-way agreement 1794, node 1762 (ID=254) isresponsible for itself as well as unoccupied identifiers 255 through 17and node 1704 (ID=44) is responsible for itself as well as identifierrange 24 through 43. Although the midway point between nodes 2201 and2202 is at unoccupied identifier 17, node 1762 is assignedresponsibility for unoccupied identifier 17 such that each unoccupiedidentifier is the responsibility of a single node.

Similarly, upon identifying node 1704 as a new immediate predecessornode, nodes 1701 and 1704 formulate cached two-way agreement 1783 thatdivides responsibility for unoccupied identifier range 1716 (unoccupiedidentifiers 45 through 64). Unoccupied identified range 1716 includesportions of unoccupied identifier range 1714, which were previously theresponsibility of node 1701. Thus, portions of unoccupied identifierrange 1714, which were the responsibility of node 1701, can become theresponsibility of node 1704 when node 1704 joins ring 1750.

Accordingly, as depicted by responsibility boundary 1773 (betweenunoccupied identifier 54 and unoccupied identifier 55), node 1704(ID=44) and node 1701 (ID=64) formulate cached two-way agreement 1783.In accordance with cached two-way agreement 1783, node 1704 (ID=44) isresponsible for itself as well as unoccupied identifiers 45 through 54and node 1701 (ID=64) is responsible for itself as well as identifierrange 55 through 63. Although the midway point between nodes 2201 and2202 is at unoccupied identifier 54, node 1704 is assignedresponsibility for unoccupied identifier 54 such that each unoccupiedidentifier is the responsibility of a single node.

During time between the joining of node 1704 and formulation of cachedtwo-way agreement 1794, nodes 1762 and 1704 do not process messagesindicated for delivery to identifiers in the range between 255 and 43.Instead, nodes 1762 and 1704 queue any messages, expect those forformulating cached two-way agreement 1794. After formulation of thecached two-way agreement 1794 is complete, nodes 1762 and 1704 can thenprocess the messages in accordance with cached two-way agreement 1794.

Similarly, during time between the joining of node 1704 and formulationof cached two-way agreement 1783, nodes 1704 and 1701 do not processmessages indicated for delivery to identifiers in the range between 45and 63. Instead, nodes 1704 and 1701 queue any messages, expect thosefor formulating cached two-way agreement 1783. After formulation of thecached two-way agreement 1783 is complete, nodes 1704 and 1701 can thenprocess the messages in accordance with cached two-way agreement 1783.

From the perspective of node 1704, the combination of cached two-wayagreement 1794 and cached two-way agreement 1783 can essentiallyrepresent a corresponding three-way agreement (not shown) between node1704, node 1762, and 1701. From the perspective of node 1704, thecorresponding represented three-way agreement defines responsibility for(assigned and unoccupied) identifiers from and including ID=254 to andincluding ID=64.

Embodiments of the invention include joining a two-way agreement.

Joining a two-way agreement can include an act of a current nodeaccessing an indication that the configuration of the ring of nodes haschanged, the indication indicative of a need to formulate a two-wayagreement dividing responsibility for at least unoccupied identifiers onthe ring between the current node and the immediate neighbor node. Forexample, referring to FIGS. 17A and 17B, node 1701 and/or node 1762 canaccess an indication, for example, from node 1702, through monitoring ofnode 1702, or from an arbitrator, that node 1702 departed ring 1750. Theindication of node 1702 departing ring 1750 indicates to node 1701and/or node 1762 a need to formulate a two-way agreement dividingresponsibility for unoccupied identifier range 1714 (unoccupiedidentifiers 255 through 63).

Alternately, referring to FIGS. 17C and 17D, node 1701 can access anindication (e.g., sent as part of the join process of node 1704) thatnode 1704 has joined ring 1750. The indication of node 1704 joining ring1750 indicates to node 1701 a need to formulate a two-way agreementdividing responsibility for unoccupied identifier range 1716 (unoccupiedidentifiers 45 through 63). Similarly, node 1762 can access anindication (e.g., sent as part of the join process of node 1704) thatnode 1704 has joined ring 1750. The indication of node 1704 joining ring1750 indicates to node 1762 a need to formulate a two-way agreementdividing responsibility for unoccupied identifier range 1715 (unoccupiedidentifiers 255 through 43).

Joining a two-way agreement can include an act of the current node andthe immediate neighbor node agreeing to a responsibility boundarybetween the current node and the immediate neighbor node that is todivide responsibility for the unoccupied identifiers between the currentnode and the immediate neighbor node. Unoccupied identifiers between thecurrent node and the responsibility boundary are the responsibility ofthe current node and unoccupied identifiers between the responsibilityboundary and the immediate neighbor node are the responsibility of theimmediate neighbor node.

For example, referring to FIG. 17B node 1701 and node 1762 can agree toresponsibility boundary 1753, which is essentially between unoccupiedidentifiers 31 and 32. Thus, unoccupied identifiers between node 1701and responsibility boundary 1753 (i.e., unoccupied identifiers 32through 63) are the responsibility of node 1701. Likewise, unoccupiedidentifiers between responsibility boundary 1753 and node 1762 (i.e.,unoccupied identifiers 255 through 31) are the responsibility of node1762.

Referring to FIG. 17D, node 1701 and node 1704 can agree toresponsibility boundary 1773, which is essentially between unoccupiedidentifiers 54 and 55. Thus, unoccupied identifiers between node 1701and responsibility boundary 1773 (i.e., identifiers 55 through 63) arethe responsibility of node 1701. Likewise, unoccupied identifiersbetween responsibility boundary 1773 and node 1704 (i.e., unoccupiedidentifiers 45 through 54) are the responsibility of node 1704.

Still referring to FIG. 17D, node 1704 and node 1762 can agree toresponsibility boundary 1793, which is essentially between unoccupiedidentifiers 17 and 24. Thus, identifiers between node 1704 andresponsibility boundary 1793 (i.e., unoccupied identifiers 24 through43) are the responsibility of node 1704. Likewise, unoccupiedidentifiers between responsibility boundary 1793 and node 1762 (i.e.,unoccupied identifiers 255 through 17) are the responsibility of node1762.

Joining and Leaving a Ring of Nodes With Ring Consistency

As previously described, in some embodiments nodes can federatedtogether to form a consistent federation infrastructure. FIG. 18illustrates a ring architecture 1800 that facilitates joining of a nodeto a ring of nodes within a federation with ring consistency. Ringarchitecture 1800 includes ring of nodes 1805. In some embodiments, ringof nodes 1805 may be similar to or the same as ring 1750 in FIG. 17C, asdescribed above. Ring of nodes 1805 may include joining node 1810 whichmay be attempting to join the ring between immediately adjacent node 1(1820) and immediately adjacent node 2 (1830). In some embodiments,joining node 1810 may join ring of nodes 1810 in a manner similar tothat described in FIG. 17C, where the joining node determines anidentifier range based on a cached agreement between nodes 1701 and1762. FIG. 19 illustrates a state diagram 1900 for a joining nodejoining the ring of nodes with ring consistency.

Embodiments of the invention include maintaining ring consistency duringthe joining of a node to a federation infrastructure, which will bedescribed with respect to FIGS. 18 and 19. Maintaining ring consistencyduring the joining of a node can include an act of a joining nodedetecting a neighborhood of a plurality of other nodes on the ring, theneighborhood including at least an immediately adjacent predecessor nodeand an immediately adjacent successor node. For example, joining node1810 may detect and/or establish a neighborhood of a plurality of othernodes on ring 1805, where the neighborhood includes immediately adjacentnode 1820 and other immediately adjacent node 1830. In some embodiments,such as in state diagram 1900, joining node 1905 may establish aneighborhood by sending introduction messages (e.g. Intro 1906) instep 1. Each node that receives such an introduction message may respondwith an acknowledgement (ACK) message (e.g. ACK 1907) in step 2 of thestate diagram. The intro 1906 may include one or more portions ofinformation used to identify the joining node and indicate that joiningnode 1805 intends to join ring 1805.

From the ACK messages received back by joining node 1905, the joiningnode may be configured to determine which node is the closest to it onthe ring. For example, each ACK message may include identifier rangesand/or position identifiers indicating the nodes position on the ringand the ranges for which the node has responsibility. Thus, in statediagram 1900, joining node 1905 may determine that immediately adjacentnode 3 (1910) is the joining node's immediately adjacent predecessornode and that immediately adjacent node 5 (1915) is the joining node'simmediately adjacent successor node. Furthermore, joining node 1905 maydetermine that adjacent node 1 (1920) and adjacent node 2 (1925) are onthe same ring as the joining node, but are not necessarily the joiningnode's immediately closest nodes. Thus, neighborhood establishment 1950may be accomplished according to exemplary state diagram 1900.

Maintaining ring consistency during the joining of a node can include anact of the joining node indicating to one of the immediately adjacentnodes selected from among the immediately adjacent predecessor node andan immediately adjacent successor node, the intent of the joining nodeto take id-space ownership for a portion of the id-space between thejoining node and the selected immediately adjacent node. For example,joining node 1810 may indicate to immediately adjacent node 1 (1820)selected from among immediately adjacent node 1 (1820) and immediatelyadjacent node 2 (1830), the intent of joining node 1810 to take id-spaceownership for a portion of the id-space between joining node 1810 andselected immediately adjacent node 1820. As previously described,id-space may include an identifier range (unoccupied or otherwise) forwhich a given node is responsible. For example, id-space may include anumerical range of node identifiers for which a given node isresponsible.

In some embodiments, such as in state diagram 1900, the act of thejoining node 1905 indicating to one of the immediately adjacent nodesselected from among the immediately adjacent predecessor node and animmediately adjacent successor node, the intent of the joining node totake id-space ownership for a portion of the id-space between thejoining node 1905 and the selected immediately adjacent node 1910comprises an act of sending a token request 1931 to a selectedimmediately adjacent node 1910 from among the immediately adjacentpredecessor node 1910 and an immediately adjacent successor node 1915,the token request including a node identifier such that only the nodewith the node identifier is capable of replying and a first time-to-liveduration value 1931, the first time-to-live duration value indicative ofa duration for which the joining node 1905 can assume a monitoringrelationship with the selected immediately adjacent node is active.

In some cases, the token request message 1931 includes a markerindicating an updated status of the joining node's 1905 expectedownership range. Time-to-live values (TTL's) and relationship monitoringmay be substantially the same as described in FIGS. 15A-15C.

Maintaining ring consistency during the joining of a node can include anact of the joining node indicating an intent to monitor the selectedimmediately adjacent node. For example, joining node 1810 may indicatean intent to monitor immediately adjacent node 1 (1820) as indicated inmonitoring indication 1812. In such a monitoring relationship, joiningnode 1810 may agree to monitor a certain range of node identifiers. Insome cases, a range may include identifiers between those of immediatelyadjacent node 1820 and immediately adjacent node 1830. In someembodiments, monitoring indication 1812 and an id-space ownershipindication may be combined in a single message.

Maintaining ring consistency during the joining of a node can include anact of a first selected immediately adjacent node receiving anindication from the joining node indicating the intent of the joiningnode to take id-space ownership for a portion of the id-space betweenthe joining node and the first selected immediately adjacent node. Forexample, immediately adjacent node 1 (1820) may receive an indication(e.g. id-space ownership indication 1811) from joining node 1810indicating the intent of joining node 1810 to take id-space ownershipfor a portion of the id-space between the joining node and node 1820.

Maintaining ring consistency during the joining of a node can include anact of the first selected immediately adjacent node receiving anindication from the joining node of the joining node's intent to monitorthe selected immediately adjacent node. For example, immediatelyadjacent node 1 (1820) may receive an indication (e.g. monitoringindication 1812) from joining node 1810 of the joining node's intent tomonitor immediately adjacent node 1820.

Maintaining ring consistency during the joining of a node can include anact of the first selected immediately adjacent node sending anindication to the joining node indicating acceptance of the joiningnode's intent to take id-space ownership for a portion of the id-spacebetween the joining node and the first selected immediately adjacentnode. For example, immediately adjacent node 1 (1820) may send id-spaceownership acceptance 1873 indicating acceptance of the joining node'sintent to take id-space ownership for a portion of the id-space betweenjoining node 1810 and immediately adjacent node 1820 (e.g. id-spaceownership acceptance 1873).

Maintaining ring consistency during the joining of a node can include anact of the first selected immediately adjacent node sending a secondindication to the joining node indicating acceptance of the joiningnode's intent to monitor the selected immediately adjacent node. Forexample, immediately adjacent node 1 (1820) may send monitoringacceptance 1883 indicating acceptance of the joining node's 1810 intentto monitor selected immediately adjacent node 1820.

Maintaining ring consistency during the joining of a node can include anact of the first selected immediately adjacent node sending a thirdindication to the joining node indicating the first selected immediatelyadjacent node's intent to monitor the joining node. For example,immediately adjacent node 1 (1820) may send monitoring indication 1893to joining node 1810 indicating the first selected immediately adjacentnode's 1820 intent to monitor joining node 1810. In some embodiments,the first, second and third indications may be combined in a singleindication. Additionally or alternatively, any combination of the first,second and third indications may be combined in a single combination(e.g. first and third or second and third).

Maintaining ring consistency during the joining of a node can include anact of the joining node receiving a first indication from the selectedimmediately adjacent node indicating acceptance of the joining node'sintent to take id-space ownership for a portion of the id-space betweenthe joining node and the selected immediately adjacent node. Forexample, joining node 1810 may receive id-space ownership acceptance1873 from immediately adjacent node 1 (1820) that indicates acceptanceof the joining node's intent to take id-space ownership for a portion ofthe id-space between joining node 1810 and immediately adjacent node1820.

Maintaining ring consistency during the joining of a node can include anact of the joining node receiving a second indication from the selectedimmediately adjacent node indicating acceptance of the joining node'sintent to monitor the selected immediately adjacent node. For example,joining node 1810 may receive monitoring acceptance 1883 from selectedimmediately adjacent node 1820 indicating acceptance of the joiningnode's intent to monitor selected immediately adjacent node 1820.

Maintaining ring consistency during the joining of a node can include anact of the joining node receiving a third indication from the selectedimmediately adjacent node indicating the first selected immediatelyadjacent node's intent to monitor the joining node. For example, joiningnode 1810 may receive monitoring indication 1893 from selectedimmediately adjacent node 1820 indicating selected immediately adjacentnode's intent to monitor joining node 1810. In some cases, the first,second and third indications may be combined in a single indication.Additionally or alternatively, any combination of the first, second andthird indications may be combined in a single combination.

In some embodiments, such as in state diagram 1900, the act of thejoining node 1905 receiving an indication from the selected immediatelyadjacent node 1910 that indicates acceptance of the joining node'sintent to take id-space ownership for a portion of the id-space betweenthe joining node and the selected immediately adjacent node andindicates the selected immediately adjacent node's 1910 intent tomonitor the joining node 1905 comprises an act of receiving a firsttoken transfer 1932 from the selected immediately adjacent node 1910,the first token transfer including the joining node's ownership range ofunoccupied node identifiers in the ring of nodes between the joiningnode 1905 and the selected immediately adjacent node 1910, a secondtime-to-live duration value 1932, the second time-to-live duration valueindicative of a duration for which the selected immediately adjacentnode can assume a monitoring relationship with the joining node 1905 isactive and a first establish grant indicative of the selectedimmediately adjacent node 1910 monitoring the joining node.

Maintaining ring consistency during the joining of a node can include anact of the joining node indicating acceptance of the selectedimmediately adjacent node's intent to monitor the joining node. Forexample, joining node 1810 may indicate in monitoring agreement 1814acceptance of the first selected immediately adjacent node 1's (1820)intent to monitor joining node 1810. In some embodiments, such as instate diagram 1900, the act of the joining node 1905 indicatingacceptance of the selected immediately adjacent node's 1910 intent tomonitor joining node 1905 comprises an act of sending an acknowledgementmessage 1933 to the selected immediately adjacent node 1910acknowledging the selected immediately adjacent node's intent to monitorjoining node 1905.

Referring again to FIG. 27, maintaining ring consistency during thejoining of a node can include an act of the first selected immediatelyadjacent node receiving an indication from joining node 1810 indicatingacceptance of the first selected immediately adjacent node's intent tomonitor joining node 1810. For example, immediately adjacent node 1820may receive an agreement (e.g. monitoring relationship agreement 1814)from joining node 1810 agreeing to participate in a one-way monitoringrelationship with immediately adjacent node 1820.

In some embodiments, selected immediately adjacent node 1820 may,additionally or alternatively, perform the acts of indicating to asecond selected immediately adjacent node the first node's intent toterminate any monitoring relationships with the second selectedimmediately adjacent node, receiving an indication from the secondselected immediately adjacent node indicating the second node's intentto terminate any monitoring relationships with the first selectedimmediately adjacent node and acknowledging the second node's intent toterminate. For example, immediately adjacent node 1 (1820) may indicateto immediately adjacent node 2 (1830) node 1's intent to terminate anymonitoring relationships with node 2 (1830). Immediately adjacent node 1(1820) may also receive an indication from node 2 (1830) indicating node2's intent to terminate any monitoring relationships with node 1.Immediately adjacent node 1 (1820) may also acknowledge node 2's intentto terminate.

In some cases, such as in state diagram 1900, immediately adjacent node3 (1910) may be configured to indicate to immediately adjacent node 5(1915) node 3's intent to terminate any monitoring relationships withnode 5 (1915) in step 5 (1934) of the state diagram. Immediatelyadjacent node 3 (1910) may also receive an indication from node 5 (1915)indicating node 5's intent to terminate any monitoring relationshipswith node 3 in step 6 (1935) of the state diagram. This may also serveto ack 1934 in some embodiments. Immediately adjacent node 3 (1910) mayalso acknowledge node 5's intent to terminate in step 7 (1936) of thestate diagram. It should be noted that the steps (1-8) of state diagram1900 may occur in series or in parallel. Thus, in some embodiments, allsteps labeled (5), for example, may occur simultaneously and others mayoccur in series. Any combination of steps performed in series orparallel is possible.

In some embodiments, maintaining ring consistency during the joining ofa node can include optional acts of receiving an introduction messagefrom the joining node. The introduction message can include an expectedownership range of node identifiers indicated as being owned by thejoining node and sending an acknowledgment message to the joining node.The acknowledgement message can include an indication that the range ofnode identifiers identified by the joining node is consistent with theselected immediately adjacent node's range of node identifiers. In somecases, the range of node identifiers may not be included in either ofthe introduction message or the acknowledgement messages. Instead, therange of node identifiers may be calculated at the time of reception orsending by the node that sent or received the message.

Maintaining ring consistency during the joining of a node can includeact of another immediately adjacent node, selected from among theimmediately adjacent predecessor node and an immediately adjacentsuccessor node, indicating to the joining node id-space ownership forthe portion of id-space between the joining node and the first selectedimmediately adjacent node. For example, immediately adjacent node 2(1830), selected from among immediately adjacent node 1 (1820) and animmediately adjacent node 2 (1830), may indicate an id-space ownershipindication 1884 to joining node 1810 id-space ownership for the portionof id-space between joining node 1810 and immediately adjacent node1830.

Maintaining ring consistency during the joining of a node can includeincludes an act of the other immediately adjacent node indicating theother immediately adjacent node's intent to monitor the joining node.For example, immediately adjacent node 2 (1830) may indicate node 2'sintent (1894) to monitor joining node 1810.

Maintaining ring consistency during the joining of a node can includeincludes an act of the joining node receiving an indication from theother immediately adjacent node that indicates id-space ownership forthe portion of id-space between the joining node and the otherimmediately adjacent node. For example, joining node 1810 may receiveid-space ownership indication 1884 from immediately adjacent node 2(1830) that indicates id-space ownership for the portion of id-spacebetween joining node 1810 and immediately adjacent node 1830.

Maintaining ring consistency during the joining of a node can include anact of the joining node receiving an indication from the otherimmediately adjacent node indicating the other immediately adjacentnode's intent to monitor the joining node. For example, joining node1810 may receive monitoring indication 1894 from other immediatelyadjacent node 1830 indicating the other immediately adjacent node'sintent to monitor joining node 1810.

In some embodiments, such as in state diagram 1900, the act of thejoining node 1905 receiving an indication from the other immediatelyadjacent node 1915 that indicates id-space ownership for the portion ofid-space between the joining node 1905 and the other immediatelyadjacent node 1915 indicates the other immediately adjacent node'sintent to monitor the joining node comprises an act of receiving a thirdtime-to-live duration value 1937, the third time-to-live duration valueindicative of a duration for which the other immediately adjacent node1915 can assume a monitoring relationship with the joining node 1905 isactive. In some embodiments, the joining node may receive a second tokentransfer 1937 from the other immediately adjacent node 1915 in step 6,the second token transfer including the joining node's ownership rangeof unoccupied node identifiers between the joining node 1905 and theother immediately adjacent node 1915. In other cases, the ownershiprange may be computed by either or both of the joining node and theother immediately adjacent node.

Referring again to FIG. 18, maintaining ring consistency during thejoining of a node can include an act of the joining node indicating tothe other immediately adjacent node the intent of the joining node toestablish id-space ownership for a portion of the id-space between thejoining node and the other immediately adjacent node. For example,joining node 1810 may indicate (e.g. in id-space ownership indication1821) to immediately adjacent node 2 (1830) the intent of joining node1810 to establish id-space ownership for a portion of the id-spacebetween joining node 1810 and immediately adjacent node 1830.

In some embodiments, such as in state diagram 1900, the act of thejoining node 1905 indicating to the other immediately adjacent node 1915the intent of the joining node to establish id-space ownership for aportion of the id-space between the joining node 1905 and the otherimmediately adjacent node 1915 comprises an act of sending anestablishment request (1938 in step 7) to establish a second ownershiprange between the joining node 1905 and the other immediately adjacentnode 1915, the establishment request 1938 including a fourthtime-to-live duration 1938, the fourth time-to-live duration indicativeof a duration for which the joining node 1905 can assume a monitoringrelationship with the other immediately adjacent node 1915 is active,and a third establish grant indicative of the joining node monitoringthe other immediately adjacent node 1915. In some embodiments, theestablishment request 198 can also represent an indication of a secondownership range between the joining node 1905 and the other immediatelyadjacent node 1915. In other cases, as explained above, the range may becalculated by any immediately adjacent pair of the nodes on the ring ofnodes 1805.

Maintaining ring consistency during the joining of a node can include anact of the joining node initiating an intent to monitor the otherimmediately adjacent node. For example, joining node 1810 may initiate(e.g. via monitoring indication 1822) an intent to monitor immediatelyadjacent node 2 (1819). In some embodiments, such as in state diagram1900, the act of the joining node 1905 indicating to the otherimmediately adjacent node 1915 the intent of the joining node toestablish id-space ownership for a portion of the id-space between thejoining node 1905 and the other immediately adjacent node 1915 comprisesan act of sending an establishment request 1938 to establish a secondownership range between the joining node 1905 and the other immediatelyadjacent node 1915, the establishment request 1938 including a secondownership range between the joining node and the other immediatelyadjacent node, a fourth time-to-live duration 1938, the fourthtime-to-live duration indicative of a duration for which the joiningnode can assume a monitoring relationship with the other immediatelyadjacent node is active, and a third establish grant 1938 indicative ofthe joining node monitoring the other immediately adjacent node.

Maintaining ring consistency during the joining of a node can include anact of the other immediately adjacent node receiving an indication ofthe joining node's intent to establish id-space ownership for a portionof the id-space between the joining node and the first selectedimmediately adjacent node. For example, immediately adjacent node 2(1830) may receive id-space ownership indication 1821 indicating thejoining node's intent to establish id-space ownership for a portion ofthe id-space between joining node 1810 and immediately adjacent node1830.

Maintaining ring consistency during the joining of a node can include anact of the other immediately adjacent node receiving an indication ofthe joining node's intent monitor the first selected immediatelyadjacent node. For example, immediately adjacent node 2 (1830) mayreceive monitoring indication 1822 indicating joining node's intent tomonitor immediately adjacent node 1830.

Maintaining ring consistency during the joining of a node can include anact of the other immediately adjacent node indicating to the joiningnode the other node's intent to monitor the joining node. For example,immediately adjacent node 2 (1830) may indicate to joining node 1810(e.g. via monitoring indication 1894) the immediately adjacent node'sintent to monitor joining node 1810.

In some cases, immediately adjacent node 2 (1830) may, additionally oralternatively, perform the acts of receiving an indication from a secondselected immediately adjacent node indicating the second node's intentto terminate any monitoring relationships with the first selectedimmediately adjacent node, indicating to the second selected immediatelyadjacent node the first node's intent to terminate any monitoringrelationships with the second selected immediately adjacent node andreceiving an acknowledgment acknowledging the first node's intent toterminate. The other immediately adjacent node may also acknowledge theindication from the second selected immediately adjacent node. Forexample, immediately adjacent node 2 (1830) may receive an indicationfrom immediately adjacent node 1 (1820) indicating node 1's intent toterminate any monitoring relationships with node 2. Node 2 (1830) mayalso receive an acknowledgement (3036 in state diagram 3000)acknowledging node 2's intent to terminate. Node 2 (1830) may alsoacknowledge the indication from node 1 (1820).

Maintaining ring consistency during the joining of a node can include anact of the joining node receiving an indication from the otherimmediately adjacent node indicating the other immediately adjacentnode's intent to monitor the joining node. For example, joining node1810 may receive monitoring indication 1894 from immediately adjacentnode 2 (1830) (e.g. monitoring) indicating the other immediatelyadjacent node's intent to monitor joining node 1810. In someembodiments, such as in state diagram 1900, the act of the joining nodereceiving an indication from the other immediately adjacent nodeindicating the other immediately adjacent node's intent to monitor thejoining node comprises an act of receiving a fourth establish grant(e.g. 1939 in step 8) for the establishment request, the fourthestablish grant indicative of the other adjacent node 1915 monitoringthe joining node 1905.

Furthermore, joining node 1810 may receive a negative acknowledge (NAK)message from at least one of the nodes on the ring (e.g. immediatelyadjacent node 1 (1820), where the NAK message includes an indication ofthe NAK sender's view of the ring. Using the NAK sender's view of theneighborhood, joining node 1810 may update its view of the neighborhoodbased on the NAK sender's view of the ring.

FIG. 20 illustrates a state diagram 2000 for a leaving node leaving thering of nodes with ring consistency. Embodiments of the inventioninclude maintaining ring consistency during the leaving of a node from afederation infrastructure, which will be described with respect to FIGS.18 and 20.

Maintaining ring consistency during the leaving of a node from afederation can include an act of the first selected immediately adjacentnode receiving an indication of the leaving node's act of leaving thering of nodes. For example, immediately adjacent node 3 (2010) mayreceive an indication from any node on the ring (e.g. from leaving node4 (2005)) indicating leaving node's 2005 act of leaving the ring ofnodes. In some cases, leaving node 2005 may already have left the ringof nodes when the indication is received. Or, leaving node 2005 may beeither in the process of leaving or planning to leave the ring of nodes.In some embodiments, the act of the first selected immediately adjacentnode 2010 receiving an indication that leaving node 2005 is leaving thering of nodes comprises the first selected immediately adjacent node2010 receiving a departure message 2021 (step 1 in state diagram 2000)from leaving node 2005, where the departure message may include anownership range of node identifiers indicated as being owned by leavingnode 2005. In other embodiments, immediately adjacent node 2010 mayreceive an indication (e.g. leaving indication 2021) from a node outsidethe ring of nodes such as from an underlying bus and/or network.

Maintaining ring consistency during the leaving of a node from afederation can include an act of the first selected immediately adjacentnode sending an indication to the second selected immediately adjacentnode indicating an intent to assume the leaving node's id-spaceownership for a portion of the id-space between the leaving node and thefirst selected immediately adjacent node (act 2920). For example,immediately adjacent node 3 (2010) may send an indication (e.g.Establish & TTL 2022) to immediately adjacent node 5 (2015) indicatingan intent to assume the leaving node's id-space ownership for a portionof the id-space between leaving node 2005 and immediately adjacent node2010.

Maintaining ring consistency during the leaving of a node from afederation can include an act of the first selected immediately adjacentnode sending an indication to the second selected immediately adjacentnode indicating establishment of at least a one-way monitoringrelationship between the first selected immediately adjacent node andthe second selected immediately adjacent node. For example, immediatelyadjacent node 3 (2010) may send an indication (e.g. Establish & TTL2022) to immediately adjacent node 5 (2015) indicating establishment ofat least a one-way monitoring relationship between immediately adjacentnode 3 (2010) and immediately adjacent node 5 (2015).

In some embodiments, such as in state diagram 2000, the act of the firstselected immediately adjacent node sending an indication to the secondselected immediately adjacent node indicating an intent to assume theleaving node's id-space ownership for a portion of the id-space betweenthe leaving node and the first selected immediately adjacent nodecomprises an act of the first selected immediately adjacent node 2010sending a first establishment request 2022 (e.g. in step 2 of statediagram 2000) to the second selected immediately adjacent node 2015 toestablish an ownership range between the first selected immediatelyadjacent node 2010 and the second selected immediately adjacent node2015, the first establishment request including a first time-to-liveduration 2022, the first time-to-live duration indicative of a durationfor which the first selected immediately adjacent node 2010 can assume amonitoring relationship with the second selected immediately adjacentnode 2015 is active, and an act of the first adjacent node 2010receiving a first establish grant 2023 (e.g. in step 3 of state diagram2000) for the first establishment request 2022, the first establishgrant 2023 indicative of the second adjacent node 2015 monitoring thefirst selected immediately adjacent node 2010.

Maintaining ring consistency during the leaving of a node from afederation can include an act of the first selected immediately adjacentnode receiving an indication from the second selected immediatelyadjacent node indicating acceptance of the first node's intent to assumeid-space ownership for a portion of the id-space between the leavingnode. For example, immediately adjacent node 3 (2010) may receive anindication (e.g. Establish & TTL 2024) from immediately adjacent node 5(2015) that indicates acceptance of immediately adjacent node 5's intentto assume id-space ownership for a portion of the id-space betweenleaving node 2005 and the first selected immediately adjacent node.

Maintaining ring consistency during the leaving of a node from afederation can include an act of the first selected immediately adjacentnode receiving an indication from the second selected immediatelyadjacent node indicating establishment of a one-way monitoringrelationship between the second selected immediately adjacent node andthe first selected immediately adjacent node. For example, immediatelyadjacent node 2010, and indicates establishment of a one-way monitoringrelationship between immediately adjacent node 2015 and immediatelyadjacent node 2010.

In some embodiments, such as in state diagram 2000, wherein the act ofan act of the first selected immediately adjacent node receiving anindication from the second selected immediately adjacent node thatindicates acceptance of the first node's intent to assume id-spaceownership for a portion of the id-space between the leaving node and thefirst selected immediately adjacent node and indicates establishment ofa one-way monitoring relationship between the second selectedimmediately adjacent node and the first selected immediately adjacentnode comprises an act of the first adjacent node 2010 receiving a secondestablishment request (e.g. in step 2 of state diagram 2000) from thesecond adjacent node 2015 to establish an ownership range between thefirst adjacent node 2010 and the second adjacent node 2015, the secondestablishment request including a second time-to-live duration 2024, thesecond time-to-live duration indicative of a duration for which thesecond adjacent node 2015 can assume a monitoring relationship with thefirst adjacent node 2010 is active, and an act of the first adjacentnode 2010 sending a second establish grant 2023 (e.g. in step 3 of statediagram 2000) for the second establishment request, the second establishgrant indicative of the first adjacent node 2010 monitoring the secondadjacent node 2015.

Data Consistency

FIG. 21 illustrates an example ring architecture 2100 that facilitatesmaintaining replica set and data consistency within a federation a ringarchitecture. In some embodiments, ring of nodes 2105 may be similar toor the same as ring 1750 in FIG. 17C and/or ring 1805 in FIG. 18, asdescribed above.

Embodiments of the invention include performing a data accesstransaction within a replica set within a federation infrastructure.

In some embodiments, performing a data access transaction within areplica set can include an act of electing a primary node from among aplurality of nodes in the replica set of nodes. For example, primarynode 2111 may be elected from among a plurality of nodes in replica setof nodes 2110. To assist in maintaining data consistency, a replica setmay be created to replicate or duplicate one or more portions of data.Thus, when nodes leave the ring (either by failure or by smoothdeparture), other nodes will also contain the information held by thedeparting node. In some cases, each node in a replica set may includethe following: 1) the identity of the replica set, 2) the configurationsequence number (CSN), 3) the membership of the replica set and who theprimary node is, 4) the last used sequence number (LSN), and 5) the datastored in the replica set. The state information represented in 1-3 maybe referred to as “configuration state” for the replica set and thestate information represented in 4-5 may be referred to as “data stored”in the replica set.

It should be understood that the principles, descriptions, andexplanations provided above regarding routing consistency and ringconsistency are applicable with respect to leader election (i.e.electing a primary node). For example, principles of maintainingid-space ownership within the ring may be applied to primary andsecondary nodes as discussed herein. In general, the properties ofprimary and secondary nodes are consistent with any or all of the othernodes described in the sections above.

In some embodiments, replica set 2110 may be created with a single nodeacting as its primary (primary node 2111). Primary node 2111 mayinitialize its LSN and CSN to zero and initialize the replicamembership, including only itself as a member. Replica set 2110 may thenbe expanded to a desired size by adding nodes to the replica set usingvarious reconfiguration algorithms, as will be described in greaterdetail below. In some cases, any node in a replica set that is not aprimary node is referred to as a secondary node. The LSN may beincremented to create a total order among both write and reconfigurationoperations. The CSN may point to the latest configuration change in thereplica set. Thus, in this case, the CSN shares the LSN sequence. Gapsin the LSN sequence typically indicate data loss, and should thus beavoided.

Performing a data access transaction within a replica set can include anact of determining that a primary node has been elected from among aplurality of nodes in a replica set of nodes, the primary node beingconfigured to accept and process client data access requests, thereplica set comprising the elected primary node and one or moresecondary nodes. For example, primary node 2111 may determine that ithas been elected from among a plurality of nodes in replica set of nodes2110, where primary node 2111 is configured to accept and process clientdata access request 2115, and where replica set 2110 includes primarynode 2111 and secondary nodes 2112.

It may be desirable to add multiple secondary nodes to increase faulttolerance (i.e. more secondary nodes storing the same data). The numberof secondary nodes to add may be determined based on a maximum number oftolerable simultaneous failures. Thus, if the maximum number oftolerable simultaneous failures is, for example, 3, it may be desirableto have at least 4 secondary nodes so that at least one node would stillbe part of the replica set even after three others have failed. In somecases, the number of secondary nodes to be members of the replica setcorresponds to the number of members in the primary node's neighborhood.

This maximum number of tolerable simultaneous failures may be related toa threshold number of secondary nodes. Such a threshold number ofsecondary nodes may be referred to as a quorum. When the data accessrequest 2115 is a data write request, the threshold number of secondarynodes may be referred to as a write quorum. Similarly, when the dataaccess request 2115 is a data read request, the threshold number ofsecondary nodes may be referred to as a read quorum. Thus, read andwrite quorums may represent a minimum threshold number of nodesnecessary to provide complete fault tolerance for a given number oftolerable simultaneous failures.

Performing a data access transaction within a replica set can include anact of the primary node receiving one or more client data accessrequests, each data access request indicating at least a portion of datathat is to be accessed on at least one of the secondary nodes in thereplica set. For example, primary node 2111 may receive client dataaccess request 2115 from client 2150, where request 2115 indicates atleast a portion of data that is to be accessed on secondary nodes 2112in replica set 2110. Data access request 2115 may, for example, includea file that is to be written to at least one of the nodes in the replicaset. Additionally or alternatively, data access request 2115 may includethe name of a file that is to be read and retrieved from at least one ofthe nodes in the replica set. The primary node 2111 manages the incomingdata requests and sends the appropriate information to the secondarynodes 2112 including configuration state information corresponding tothe replica set. The portion of data may also include a set of controlinformation about the replica set itself including a node's datasequence number (DSN), LSN, CSN, primary node identity or secondary nodeidentity.

Performing a data access transaction within a replica set can include anact of the primary node assigning a data sequence number (DSN) to eachof the client data access requests that mutate state in the order theclient data access requests are received, the data sequence numberscomprising a linearized processing order that is to be followed by eachof the nodes in the replica set. For example, primary node 2111 mayassign a DSN to each client data access request 2115 that mutate statein the order the requests are received. The DSN's may include alinearized processing order that is to be followed by each of the nodesin replica set 2110. In some cases, the linearized processing order maybe referred to as sequence of operations 2201, as illustrated in FIG.22. In cases where the request does not mutate state, primary node 2111may omit assigning a data sequence number to the client data accessrequest.

Sequence of operations 2201 may include multiple operations arranged inthe order they were received, according to a data sequence number (DSN3505). For example, primary node 2111 may assign OP₁ to represent afirst operation, OP₂ to represent a second operation, and down the lineas represented by OP_(n). OP_(n+1) may represent the last-used sequencenumber (LSN 3515). As mentioned above, LSN 2215 may be incremented tocreate a total order among both write and reconfiguration operations.The CSN 2210 may point to the latest configuration change in replica set2110. In some cases, CSN 2210 may share the LSN sequence. Gaps in theLSN sequence typically indicate that the sequence of operations 2201 hasnot been followed, which often results in data loss.

Performing a data access transaction within a replica set can include anact of the primary node sending at least one of the client data accessrequests including any corresponding data sequence numbers to at leastone of the secondary nodes. For example, primary node 2111 may sendclient data access request 2115 including any assigned DSN's 2116 tosecondary nodes 2112. In some cases, client data access request may besent to a quorum such as a write quorum or a read quorum. The quorum mayrespond with the expected information and additionally with anindication of other data access requests the primary node is unaware ofFor example, a secondary node may have processed other transactionsbefore primary node 2111 was elected or may have been assigned certaintransactions that the primary is not aware of Thus, this information maybe sent to primary node 2111, which may be used to update the primarynode's sequence of operations 3501. In some cases, this process onlyoccurs during a replica set reconfiguration, as will be explained ingreater detail below.

Performing a data access transaction within a replica set can include anact of the primary node receiving, from at least a threshold number ofsecondary nodes, an acknowledgement indicating reception of the clientdata access request. For example, primary node 2111 may receive, from athreshold number of secondary nodes (e.g. a read or write quorum), anacknowledgement message 2120 acknowledging reception of client dataaccess request 2115. The acknowledgement 2120 may further indicate thatthe secondary node that sent the message has inserted the client dataaccess request 2115 into its data access queue (i.e. sequence ofoperations 2201). In some cases, primary node 2111 may receive such anacknowledgement message from all of the nodes in the replica set or froma quorum thereof. The acknowledgement may, additionally oralternatively, indicate acceptance or rejection of the client dataaccess request.

Performing a data access transaction within a replica set can include anact of the primary node committing the data access request, thecommitting including accessing the data according to the client dataaccess request. For example, primary node 2111 may commit the dataaccess request 2115 in a single phase, where the committing includesaccessing the data according to client data access request 2115. Thus,in cases where the data access request is a write request, primary node2111 may commit the write request in a single phase. Committing in asingle phase may be done without sending and receiving commit queriesfrom each of the secondary nodes. Committing in a single phase may bereferred to as a non-blocking transaction. Because failure detection, asdescribed above, within the replica set is perfect, the primary nodedoes not have to send and receive commit queries from each of thesecondary nodes.

In some embodiments, primary node 2111 may send client data accessrequest 2115 to a second replica set. The second replica set may be in adifferent neighborhood than the neighborhood of the primary node. Insome cases, the primary node's neighborhood includes the same nodes asthose in the replica set. Primary node 2111 may request that replica set2110 handle a first portion of the client data access request 2115 andthe request the second replica to handle a second different portion ofthe data access request. Such embodiments, however, would use atwo-phase commit process. For example, the process may include receivinga reply from replica set 2110 corresponding to the first portion of theclient data access request 2115. The process may also include receivinga reply from the second replica set corresponding to the seconddifferent portion of the data access request. The replies may becombined and, based on the combined replies, an up-to-date data accessresponse may be generated.

In an exemplary embodiment, primary node 3211 may respond to a clientdata write request as follows: 1) primary 3211 validates the requestagainst the application consistency constraints. If primary 3211determines a violation, it responds to the client 3250 with a failurestatus of application consistency violation and skips to step (7) below.2) Primary 3211 assigns the write operation a sequence number which maybe obtained by atomically incrementing the primary's LSN. In thisexample, the sequence number associated with a given write operation isreferred to as its DSN. 3) The primary 3211 communicates the DSN, CSN,and data to be written to all the secondary nodes and waits for a writequorum of secondary nodes to respond (a variant approach may be for theprimary to send the data to be written to only a write quorum).

4) After hearing acknowledgements from a write quorum of secondarynodes, primary 2111 performs the write operation locally and is thepoint at which the write operation is considered committed. Primary 2111responds back to the client with a success status and skips to step (7)below. 5) If primary 2111 fails to get acknowledgements from a writequorum of secondary nodes, it initiates a reconfiguration operation toeliminate the failed secondary nodes. 6) If primary 2111 fails toperform the write operation locally, it initiates a reconfigurationoperation to transition to a new primary, removes itself from thereplica set 2110, and skips step (7). 7) Primary 2111 waits for the nextincoming client data access request. It should be noted that otherembodiments are possible and may include more or less steps, asexplained above.

In an exemplary embodiment, one or more secondary nodes 2112 may processa data write request in the following manner: 1) the secondary comparesthe CSN specified in the incoming write operation to its CSN. If thespecified CSN is less than its CSN, the secondary ignores the requestand skips to step (4) below. Secondary node 2112 asserts that thespecified CSN cannot be greater than its CSN because such a conditionimplies that reconfiguration was accomplished with an imperfect failuredetector. 2) Secondary node 2112 compares the DSN of the incoming writedata request with its LSN. If the DSN is less than its (LSN+1),secondary node 2112 ignores the write data request. If the DSN isgreater than its (LSN+1), secondary node 2112 buffers the data writerequest and skips to step (4) below. Otherwise, secondary node 2112atomically increments it's LSN and accepts the data write request, andsends back an acknowledgement 2120. If secondary node 2112 is unable toaccept the request due to local failures, it requests the primary toinitiate a reconfiguration to remove it as secondary from the replicaset. 3) Secondary node 2112 examines the currently buffered operationsand processes them until it has examined each of them, accepting anywhose DSN is within the scope of the secondary's LSN value describedhere, thus preserving the total order established by the primary. 4)Secondary node 2112 waits for the next data access request from primary2111. The act of processing can include writing the related data to astable storage facility.

In a database setting, an exemplary method for performing a writetransaction is as follows: 1) For operations prior to a “commit”including “abort”, primary 2111 simply performs the operations locallyand skips to step (6) below. 2) For “commit”, primary 2111 determinesthe transaction order by locally preparing the transaction. This stepalso validates the transaction against database consistency constraints.If the local prepare fails, it responds back to client 2150 with afailure status and skips to step (6) below. 3) Primary node 2111forwards the transaction with the “commit” request to the secondarynodes 2112 as specified in steps (2)-(3) in the previous example. 4)After primary node 2111 gets acknowledgments from a write quorum ofsecondary nodes, it commits the transaction locally and reports successto the client 2135. 5) If primary node 2111 fails to getacknowledgements from a write quorum of secondary nodes, it aborts thelocally prepared transaction and initiates a reconfiguration operationto eliminate the failed secondary nodes. 6) Primary 2111 waits for thenext incoming client data access request 2115.

Embodiments of the invention include establishing and maintaining aconsistent replica set within a federation infrastructure.

Establishing and maintaining a consistent replica set within afederation infrastructure can include establishing a replica set fromamong a plurality of nodes in a ring of nodes, the replica setcomprising as members a primary node and one or more secondary nodes.For example, replica set 2110 may be established from among nodes 2111,2112 and other nodes on ring of nodes 2105. Replica set 2110 may includeas members primary node 2111 and secondary nodes 2112. In some cases,the boundaries of replica set 2110 may be the same as the neighborhoodof the primary and which secondary nodes are also members.

Establishing and maintaining a consistent replica set within afederation infrastructure can include an act of receiving an indicationindicating a replica set configuration event occurrence affecting atleast one of the replica set members. For example, any of nodes 2111 and2112 in replica set 2110 may receive an indication of a replica setconfiguration event occurrence that affects at least one of primary node2111 and secondary nodes 2112. Replica set configuration events mayinclude a node failure within the replica set, a node departure from thereplica set or a node joining the replica set. Other events affectingnodes of the replica set that would cause a reconfiguration are alsopossible. In some cases, such a configuration event may occur during aclient data access request.

Establishing and maintaining a consistent replica set within afederation infrastructure can include includes an act of, based on thereceived indication, an act of determining that the replica set is to bereconfigured, the reconfiguration including at least one of removing anexisting secondary node from the replica set, adding a new secondarynode to the replica set, abruptly transitioning to a new primary andsmoothly transitioning to a new primary, based on the replica setconfiguration event occurrence. For example, based on the receivedindication, one of the nodes in replica set 2110 may determine thatreplica set 2110 is to be reconfigured. The reconfiguration may includeremoving an existing secondary node from the replica set, adding a newsecondary node to the replica set, abruptly transitioning to a newprimary or smoothly transitioning to a new primary, depending on whichconfiguration event occurred.

Example methods and algorithms for reconfiguring a replica set andtransitioning to a new primary node are included below. The examples ofreplica set reconfiguration generally cover four cases including 1) anabrupt transition to a new primary, 2) a smooth transition to a newprimary, 3) removing a secondary node, and 4) adding a secondary node.Each of these four general examples may occur independently and, atleast in some embodiments, do not involve replacement of an existingprimary.

Establishing and maintaining a consistent replica set within afederation infrastructure can include an act of reconfiguring thereplica set according to the reconfiguration corresponding to thereplica set configuration event. For example, replica set 2110 may bereconfigured according to the reconfiguration corresponding to thereplica set configuration event. Thus, in cases where the replica setconfiguration event is a node failure, replica set 2110 may bereconfigured by adding a secondary, removing a secondary ortransitioning to a new primary. Similarly, when the replica setconfiguration event is a node joining, replica set 2110 may bereconfigured according to any of the above methods. In some cases,certain reconfiguration methods may correspond to certainreconfiguration events. This correspondence may be configured by a useror may be automatic.

In some cases, primary node 2111 may detect, based on the configurationstate information, that one or more replica set reconfigurations inprogress. As indicated above, during the reconfiguration process, a newprimary node may be elected for the replica set. According to theembodiments described above, nodes in the ring 3205 may communicate withthe new primary node to ensure that the new primary node's informationis up-to-date with respect to the other nodes on the ring.

In some embodiments, a consistent replica set similar to or the same asthe one described above, may be used to access and store data. Forexample, replica set 2110 may be configured to maintain a portion ofdata storage on one or more of the replica set nodes (e.g. on secondarynodes 2112). Primary node 2111 may receive client data access request2115. Request 2115 may include an indication of information that is tobe accessed on at least one node within replica set of nodes 2110.Primary node may either process the request directly, or pass it to asecondary node. In cases where primary node 2111 detects that it hasfailed to access at least a portion of information indicated in thereceived client data access request 2115, primary node 2111 may removeitself from the replica set so that information stored on primary node2111 is prevented from being accessed in response to the client dataaccess request. This eliminates the possibility of returning stale orout-of-date information and ensures that any information returned inresponse to client data access request 2115 is up-to-date.

According to some embodiments, an abrupt transition to a new primary mayoccur in the following manner. When the existing primary fails, a newprimary is elected using reliable leader election algorithm per themethod mentioned above. Before accepting any client operations, the newprimary may perform the following configuration change to remove the oldprimary from the replica set: 1) The new primary checks to see if it hasknowledge of the replica set state. If the new primary does not have theknowledge, it impliedly determines that all members of the replica sethave failed and it skips to step (9) below. 2) The new primary contactsevery node of the replica set to determine their CSN and LSN until ithits a fix point in terms of having contacted all existing secondarynodes. Using the information collected in this step, the new primarydetermines the earliest CSN, latest CSN, and latest LSN. Note that theearliest CSN and latest CSN may be same in some (or most) cases. Whenthe earliest and latest CSN are different, failure of primaries in themiddle of reconfiguration operations is indicated. Then, any replicawith a CSN larger than the earliest CSN definitively has the latestdata. Thus, at the end of this step, the new primary already has thelatest data when the earliest CSN and latest CSN are different.Otherwise, the new primary only has data written prior to latest CSN andthe check in step (3) ensures that it also has the data written past thelatest CSN.

3) If the earliest CSN and latest CSN is the same and the new primaryhas failed to contact a read quorum of secondary nodes, there is apotential for data loss and the new primary skips to step (9) below. Insome cases, it may be highly advantageous (or even necessary) for theread quorum to intersect with its write quorum. For size based quorums,the read quorum is any set larger than (n-w) where n is the number ofreplica nodes specified in the CSN and w is the write quorum.

4) The new primary attempts to bring the secondary nodes in itsconfiguration current with respect to data by sending them theoperations between their reported LSN and latest LSN determined in step(2) above. Incomplete reconfiguration operations after the earliest CSNand up to latest CSN may be sent as abbreviated void reconfigurationoperations in this step. 5) The new primary builds a new replicaconfiguration locally after removing the failed primary (e.g. primary2111) and secondary nodes (e.g. secondary nodes 2112) from theconfiguration with the latest CSN, makes the latest LSN its LSN, andassigns the new configuration a CSN obtained by incrementing the LSN. 6)The new primary communicates the new replica configuration to all thesecondary nodes.

7) If the new primary fails to get success status from all secondarynodes or hears a failure status from any secondary in response to thereconfiguration operation sent in step (6) above, the new primaryre-executes steps (2)-(6) above. 8) After hearing success status from atleast some of (or all) the secondary nodes, the new primary skips tostep (10) below. 9) The new primary takes steps to perform disasterrecovery such as ringing bells in the data center, sending pagermessages, etc. In some cases, the replica set can only be restored usingmanual intervention. 10) The new primary resumes processing of theclient operations at this point. In step (2) above, the new primarymight discover new secondary nodes that were in the process of beingadded to the replica set when the old primary failed. The fix point ismeant to capture the stable set of secondary nodes that the new primaryconsiders part of the new configuration. If we eliminate theoptimization around the abrupt primary transition also handling newsecondary nodes, this fix point computation may be eliminated.

According to some embodiments, a smooth transition to a new primary mayoccur in the following manner. The transition from an existing primaryto a new primary is very similar to adding a new secondary to thereplica set (as will be explained below). Note that the new primary mayor may not be a current secondary in the replica set. In this example,the potential new primary follows this algorithm to become part of thereplica set: 1) the new primary contacts one or more existing secondarynodes in the replica set to transfer data to it, 2) the new primarysimultaneously contacts the existing primary (e.g. primary 2111) toforward new operations to it.

3) If the DSN of the new operation received from the new primary islarger than the data received from a secondary, it merges the new writeoperations sent by the new primary with the data received from thesecondary. 4) After all the secondary nodes contacted by the new primaryin step (1) above report that the data transfer is complete, the newprimary contacts the existing primary 2111 to initiate a configurationchange to make the new primary the current primary of the replica set.5) After receiving success status from the existing primary as aresponse to the reconfiguration request sent in step (4), the newprimary assumes the primary role for the replica set, and 6) until thereconfiguration request in step (5), the new primary forwards any clientoperations it receives to the existing primary.

According to some embodiments, the process of removing a secondary nodemay occur in the following manner. When a perfect failure detector (asdescribed above with regard to FIGS. 14 and 19) reports to the one ofthe ring nodes (e.g. primary node 2111) that an existing secondary 2112has failed or when an existing secondary contacts one of the other nodes(e.g. the primary) to voluntarily remove itself from the replica set2110, one of the nodes (e.g. the primary) may exercise the followingalgorithm to remove the secondary from the replica set. In thisexample, 1) the primary blocks processing of incoming client operations(e.g. 2115) and either buffers them or asks the clients (e.g. 2150) toretry later, 2) the primary installs a new replica configuration locallyby removing the failed secondary from the replica set membership andassigns the new configuration a CSN by atomically incrementing it's LSN.In some cases, this means that reconfiguration operations are in thesame sequence stream as data operations. 3) The primary communicates thenew configuration to all the secondary nodes, and 4) after hearingsuccess status from at least some of (or all) the secondary nodes, theprimary resumes processing of the client operations.

During such a reconfiguration process, the secondary being reconfiguredmay behave as follows: 1) the secondary compares the CSN of thereconfiguration operation with its LSN. If the CSN is less than (LSN+1),the secondary sends the primary a failure status and skips to step (3)below. If the CSN is equal to (LSN+1), the secondary atomicallyincrements it's LSN and accepts the reconfiguration operation, and sendsback a success status. Otherwise, the secondary buffers thereconfiguration operation and skips to step (3) below, 2) the secondaryexamines the currently buffered operations and processes them until ithas examined at least some (or all) of them, and 3) the secondary waitsfor the next operation request from the primary.

The perfectness of the failure detector ensures that old and newreconfigurations do not simultaneously exist. In some embodiments, anoverall assumption is that there exists a perfect failure detector thatcan be used to detect fail-stop failures. The use of “perfect” in thissense captures the assumption that the failure detector never reportsthat a node instance has become unavailable unless it (that instance ofthe node's existence) has indeed become permanently unavailable. Asmentioned above, failure detection may go hand-in-hand with and may bereliant on the concepts of ring consistency and routing consistencyproperties, as explained above. Because old and new reconfigurations donot exist simultaneously, this implies that replica set state does notexist outside of the replica set members because a replica set node isremoved from the replica set only after either it has failed orvoluntarily removed itself This property is useful in identifying thecondition where all the members of a replica set have failed because thenewly elected primary will realize that it does not have access to thereplica set state under this condition.

In this example, because the primary blocks processing of incomingoperations until the reconfiguration operation is complete, it allows anoperator to ignore the reconfiguration operations that have becomeinvisible. A reconfiguration operation may become invisible when theprimary that initiated it fails during the reconfiguration processitself and the secondary nodes that were contacted by the failed primaryalso fail subsequently. This property allows a newly elected primarythat finds a single CSN among the existing secondary nodes of thereplica set to be assured that either the CSN it found is indeed thelatest CSN from an oracle perspective observing the entire system orthat no data operations were initiated after the later reconfigurationoperations that have since become invisible. As such, it is safe for thenewly elected primary to determine data loss based on the latest CSN itfound among the existing secondary nodes as explained in the exampleregarding abrupt transition to a new primary.

Given that the reconfiguration operations can be in the same sequence asdata operations, at least some (or all) secondary nodes are broughtcurrent with respect to data (meaning they all have the same LSN) at theend of a reconfiguration operation. In the presence of a write quorum,this property makes it safe for the newly elected primary to determinedata loss based on the latest CSN it finds among the existing secondarynodes.

According to some embodiments, the process of adding a secondary nodemay occur in the following manner. In this example, the new secondaryfollows the following algorithm to become part of the replica set: 1)the secondary node contacts one or more existing secondary nodes in thereplica set to transfer data to it, 2) the secondary node simultaneouslycontacts the primary (e.g. 2111) to forward new operations to it, 3) ifthe DSN of the new operation received from the primary is larger thanthe data received from a secondary, the secondary node merges the newwrite operations sent by the primary with the data received from thesecondary, and 4) after at least some of (or all) the secondary nodescontacted by it in step (1) above report that the data transfer iscomplete, the secondary node contacts the primary to initiate aconfiguration change to add itself to the replica set.

Continuing this example, the algorithm followed by the primary to add anew secondary to the replica set is as follows: 1) in response to thefirst request from the new secondary, the primary starts forwarding newoperations to the would-be new secondary though it is not yet part ofthe replica set, and 2) in response to the subsequent reconfigurationrequest from the new secondary, the primary can initiate areconfiguration operation using virtually the same steps mentioned abovein the “removing a secondary node” example with the difference beingthat the new configuration adds the new secondary instead of removing anexisting secondary.

FIG. 23 illustrates an exemplary system 2300 for accessing data from areplica set within a federation infrastructure. Environment 2300includes data access system 2310 that comprises a plurality of modules.In general, each module described below exists within the ringimplementation (e.g. ring of nodes 2105) and may be implemented as adistributed algorithm across the nodes making up a neighborhood and/orreplica set and more generally nodes making up a ring of nodesarrangement and devices attempting to become an active node in such aring of nodes arrangement. The system includes a neighborhoodestablishing module 2320 configured to establish a neighborhood of aplurality of nodes on the ring, the neighborhood including at least animmediately adjacent predecessor node and an immediately adjacentsuccessor node. Ring of nodes 2105 may include joining node 2306 whichmay be attempting to join the ring between immediately adjacent node 1(2308) and immediately adjacent node 2 (2309). In some embodiments,joining node 2306 may join ring of nodes 2305 in a manner similar tothat described in FIG. 17C, where the joining node determines anidentifier range based on a cached agreement between nodes 1701 and1762.

The system further includes an intent determining module 2325 configuredto determine that a joining node outside the established neighborhoodhas indicated an intent to join the established neighborhood in the ringof nodes. Intent determining module 2325 may receive neighborhoodindication 2321 from neighborhood establishing module 2320 indicatingthe establishment of a neighborhood of nodes. The neighborhood mayinclude immediately adjacent node 1 (2308), joining node 2306 andimmediately adjacent node 2 (2309). Intent determining module 2325 maysend joining node's intent to join 2326 to consistency maintainingmodule 2330.

Consistency maintaining module 2330 may be configured to maintainconsistency within the established neighborhood such that each node inthe plurality of nodes takes id-space ownership for a portion ofid-space in the neighborhood. As described above under the heading“Joining and Leaving a Ring of Nodes,” consistency within theneighborhood may be maintained when joining node 2306 joins the ring ofnodes 2305. Consistency maintaining module 2330 may send consistencyindication 2331 to election determining module 2335. It should be notedthat, in some embodiments, for all state consistently maintained at agiven id in the id-space of a ring, there is a replica set formed forthat unit of state who's primary node is the ring node owning that id atany given moment. As id-space ownership transfers (consistently) for agiven id between nodes, so does the corresponding primary-ness of anyunits of service state stored at (or controlled by) that owning node. Inthis embodiment, the id-space ownership and primary-ness is transferredin a simultaneous manner.

Election determining module may be configured to determine that one ofthe nodes in the plurality of nodes has been elected to be a primarynode with responsibility over a replica set of nodes configured torespond to client data access requests as directed by the primary node.As explained above with regard to primary node election, electiondetermining module may determine that a primary node has been electedfor a replica set (which may include all the nodes in the establishedneighborhood) and send an indication (2336) of which node was elected tobe primary to request receiving module. As explained above, such leaderelection may follow the principles of ring consistency and routingconsistency (including monitoring agreements forming at least in part aperfect failure detector mechanism) as defined earlier in this paper.

Request receiving module 2340 may be configured to receive one or moreclient data access requests from client 2315, where each data accessrequest 2341 indicates at least a portion of data that is to be accessedon at least one of the nodes in the replica set. Request receivingmodule 2340 may also receive primary node indication 2323. The primarynode indication 2323 and the data access request 2341 may be combined2341A & 2323A (or may each be sent separately (not shown)) to dataaccessing module 2345. Data accessing module 2345 may be configured tocommunicate with one or more of the nodes in ring of nodes 2305 viacommunications 2307 and access the data indicated in the client dataaccess request 2341. Thus, an entire system may be instantiated wherenodes may join and leave ring of nodes 2305, consistency will bemaintained, primary nodes elected and data reliably accessed. Each ofthe modules described in this system may perform these functions as wellas other functions described in greater detail in the above sectionsthat correspond to each module.

In some embodiments, the data that is to be accessed comprises aservice. For example, joining node 2306 may provide a service thatclient 2315 desires to access. For example, the service may be an emailapplication. Client 2315 may indicate data that is to be accessed anddata access system 2310 may be configured to access the data, regardlessof whether the joining node leaves the ring due to the ring and dataconsistency maintained by the data access system. In other cases, theservice provided may be a lock manager, a data storage system,producer-consumer queues or any other service. In cases where the systemis used to access data such as a database, system 2310 may be configuredto maintain each of the atomicity, consistency, isolation, anddurability (ACID) properties for the data in the replica set. Thus, asystem is presented that may be configured to provide consistent,reliable data access, regardless of which node actually contains thedata, which nodes are joining or leaving the ring of nodes, or whether asubstantial number of nodes containing the data have failed.

In some cases, the operations exposed by a service can be partitionedinto those that mutate state and those that do not mutate state. In thecase of a storage service, operations that mutate state may be referredto as “writes” and operations that do not mutate state are referred toas “reads.” In the case of a lock manager service, the operations thatmutate state are referred to as “acquires” and “releases” and operationsthat do not mutate state are referred to as “queries.” In the case ofproducer-consumer queues (sometimes used for distributed computations),operations that mutate state are referred to as “enqueue” and “dequeue”and operations that do not the state are referred to as “peeks.” Thus,in some embodiments herein, the term “write” captures any operation thatmutates state and the term “read” captures any operation that does notmutate state.

Referring back to FIG. 21, in some cases, read operations are processedlocally by the primary node 2111. If a read can't be satisfied at awould-be primary because the client-requested state does not existthere, that state will not (guaranteed) exist at any of the secondarynodes. In some embodiments, write operations may be ordered thoughprimary node 2111. In such cases, primary node 2111 may not actuallywrite the client-supplied state locally until at least a write quorum ofsecondary nodes have acknowledged that they have written that statethemselves. Then, primary node 2111 actually writes (or attempts towrite) the state locally. Reconfiguration, in some embodiments, mayoccur any time a replica set member (primary or secondary node) cannotwrite the requested state. In such cases, the node may initiate areconfiguration in the replica set by removing itself from the replicaset. Furthermore, any secondary nodes not responding will fail (this isguaranteed because of the perfect failure detector) and the primary nodewill trigger a reconfiguration by removing that secondary node from thereplica set. In some cases, this will not cause the primary node to stopbeing the primary for the replica set.

Data access system 2310 may be configured to write data according to aclient data write request. Thus, in cases where data access request 2341(or data access request 2115) is or includes a write request, primarynode 2111 may instruct a secondary node 2112 to write the data indicatedin the client data write request to a data store in the secondary node.In some cases, the primary node replicates the write request to one ormore secondary nodes 2112 in two phases. The advantage of this variantis that it allows for read operations to be serviced by a secondary ifthe write quorum consists of all the secondary nodes in the replica set2110. In some cases, accepting a write operation at secondary nodes andthe primary node can trigger additional logic such as writing the datato stable storage available to each of the nodes individually.

Data access system 2310 may be configured to read data according to aclient data read request. Thus, in cases where data access request 2341(or data access request 2115) is or includes a read request, eitherprimary node 2111 or secondary nodes 2112 may be configured to handlethe read request directly. In some cases, it is unnecessary for primarynode 2111 to call on a secondary node to read the data indicated in therequest. In other cases, the secondary may be able to read the data andrespond to the client data read request autonomously.

Namespaces

As previously described, namespace management services can be used toorganize resources into collections, potentially providing resourceaccess through a plurality of namespaces. Namespaces can be viewed as aforest where each namespace (tree) is represented as a Uniform ResourceIdentifier (“URI”) with scheme and the part immediately following itserving as the root. The URI scheme can be hierarchical or flat.Hierarchical schemes such as “name” and “http” (as opposed to flatschemes such as “uuid”) can be identified by the presence of the “:/”character sequence after the scheme name. The first part of hierarchicalschemes can identify the naming authority responsible for the rest ofthe URI components. Such URIs are identified by the presence of the“://” character sequence after the scheme name. Namespaces can be bothhierarchical and are routable meaning that namespaces serve asidentifiers that can be used to identify the communication paths fromthe sender to receiver.

In some embodiments, a namespace can be defined as follows:

Namespace := Flat | Hierarchical Flat := Scheme ‘:’ Opaque_partHierarchical := Scheme “:/” (‘/’ Authority ‘/’)? Segment (‘/’ Segment)*Scheme := as defined by RFC-2396 on URI Generic Syntax Opaque_part := asdefined by RFC-2396 on URI Generic Syntax Authority := as defined byRFC-2396 on URI Generic Syntax Segment := as defined by RFC-2396 on URIGeneric Syntax

Resources can be made available at any branch in the tree, and a givenresource can be exposed in multiple namespaces. Also, a given namespacecan identify a single resource or a namespace branch (a group ofresources). Such a grouping can be logical or physical depending on thesemantics of the Namespace. The group is obtained by performing adepth-first-search on the identified Namespace branch. Once a group ofresources have been identified, many operations can be performed on themsuch as selecting resources that satisfy some criteria, sending (andpotentially routing) a given message to only those in a group, and soon.

A single resource can be viewed as a trivial collection. Thus, everyresource can be assigned a name(space). Because namespaces are routable,messages can be routed over a namespace federation infrastructure to anyresource that has a name. Such routing can cross trust boundaries andtraverse firewalls.

Generally, a resource can be assigned one or more URIs that can be usedto access the resource. One URI, the Resource ID, assigned to a resourcecan be, at a minimum, unique across all namespaces implemented by agiven namespace federation infrastructure such that the resource can besingularly referenced. Other, potentially non-unique, URIs can also beassigned to resources. These other, potentially non-unique, URIs provideaccess to the resource via additional locations within namespacesimplemented by a given namespace federation infrastructure. A resourcecan be assigned at least one potentially non-unique URI for eachnamespace that can be traversed to access the resource.

FIG. 28 illustrates an example of a namespace federation infrastructure2800 from provider and collection views of a namespace. Namespacefederation infrastructure 2800 depicts that providers can be registeredat any branch in a namespace tree. Further, a provider can be registeredat multiple namespace branches, potentially in different trees. Forexample, provider 501 is registered for the namespace brancheslocation:/CorporateBuildings/bldg34,location:/CorporateBuildings/bldg50/floor2, andlocation:/CorporateBuildings/bldg50/floor1/room1304. Provider 2802 isregistered for namespace branches location:/CorporateBuildings/bldg50and location:/CorporateBuildings/bldg26. Provider 2803 is registered forlocation:/CorporateBuildings/bldg50/floor1.

As depicted in FIG. 28, applications can view namespaces as a logicalcollection of resources that can nest hierarchically. That is,intermediate namespace nodes (e.g.,location:/CorporateBuildings/bldg50/floor1 andlocation:/CorporateBuildings/bldg50) are viewed as resources—namespacenode resources. Applications can efficiently operate on such logicalcollections in a coherent and scalable manner, including publishing,searching, locating, tracking, targeting, and sourcing events frominside the collections. Note that not all the resources inside a logicalcollection are necessarily located on a single computer system ordevice. Resources can be distributed both in space and time across manycomputer systems and device. The namespace federation infrastructuretakes care of efficiently routing lookup requests to the computersystems and devices participating in any given collection, therebyproviding a uniform and consistent view to applications.

FIG. 29 illustrates an example namespace federation infrastructure 2900with a resource made available in a plurality of namespaces. The URIOrganization:/Product identifies the root of namespace tree 2901.Similarly, URI Location:/Bldg 42 identifies the root of namespace tree2902. As depicted, printer 603 is exposed in both namespace tree 2901and namespace tree 2902.

Within this specification and the following claims, a namespace noderesource can be viewed simply as a node in a namespace tree. Somenamespace node resources can be viewed as root nodes (e.g.,Location:/Bldg42), others can be viewed as intermediate nodes (e.g.,Organization:/Product/Devices Team, and others can be viewed as leafnodes (e.g., Location:/Bldg 42/Floor 1/Room 112/Printer 2903). However,it should be understood that a namespace node resource in one namespacetree can reference a namespace node resources (or other resources) inanother namespace tree. Thus, viewing a namespace node resource as aroot, intermediate, or leaf in one namespace tree does not limit thequerying of that namespace node resource from other namespace trees.

A namespace also includes namespace segments that link (or relate) twoor more namespace node resources. A namespace segment can be utilized tolink namespace node resources in the same namespace. For example, thenamespace segment 2911 (“Devices”) links Organiaztion:/Product toDevices Team. Further, a namespace segment can link (otherwiseconnected) namespace node resources in different namespace trees therebyproviding the functionality of symbolic links. Traversing a namespacesegment includes navigation to all the target namespace node resources.For example, the namespace segment 2941 (“Project”) connects the PM Teamto the file resources SpecTemplate.doc and Milestone.prj.

Accordingly, namespace segment 2911 (“Devices”), namespace segment 2921(“Dev”), and namespace segment 2931 (“Printer”) can be traversed innamespace tree 2901 to identify printer 2903. Likewise, namespacesegment 2912 (“Floor 1”), namespace segment 2922 (“Room 1226”), andnamespace segment 632 (“Printer”) can be traversed in namespace tree2902 to identify printer 2903. It should be understand that the URIscheme of namespace tree 2901 and the URI scheme of namespace tree 2902can differ.

Since the same resource can participate in multiple namespaces and withthe presence of the symbolic link functionality, a global view of allthe namespaces and the resources participating in them forms a directedgraph with the Namespace segments serving as labeled graph edges andnamespace node resources and other resources serving as graph nodes. Thenamespace roots effectively partition the namespace node resources andother resources in this global graph into a set of starting andreachable resources with the starting namespace node resources providingthe basis for namespace scoping. Accordingly, cached information forimplementing queries is reduced and distributed across each namespace.

Also, any given namespace can form a graph since the same resource canbe made available at multiple Namespace branches and some segments canconnect otherwise connected namespace node resources.

FIG. 24 illustrates an example of a namespace federation infrastructure2400. The namespace federation infrastructure 2400 includes namespacemanagers 2401, 2402, 2403, 2411, and 2412 that can form different typesof federating partnerships. For example, namespace managers 2401, 2402,2403 are federated among one another as peers without a root namespacemanager. On the other hand, namespace managers 2411 and 2412 federatewith namespace managers 2401 and 2402 respectively with namespacemanagers 2401 and 2402 serving as root namespace managers. Differenttypes of devices can participate in the namespace federationinfrastructure, including hosts (e.g., PCs hosting resources), messagerouters, message gateways (e.g., firewalls, network address translation(“NAT” boxes, and redirectors), and message brokers (e.g., pub-subintermediaries). Namespace federation infrastructure 2400 facilitatesbus protocols (e.g., liveness, control, eventing, and streaming).Further, namespace federation infrastructure 2400 can interoperate withthird-party software and hardware stacks using related WS protocols suchas, for example, WS-Discovery and WS-Eventing.

Generally, the namespace managers 2401, 2402, 2403, 2411, and 2412 canutilize namespace federation protocols to form partnerships and exchangenamespace information. The formation of partnerships and exchange ofnamespace information facilitates more efficient and reliable access tonamespace resources. It may be that peer namespace managers (e.g.,namespace managers 2401, 2402 and 2403) exchange namespace informationwith other peer namespace manages. However, other namespace managers(e.g., namespace managers 2411 and 2412) may exchange namespaceinformation with corresponding root namespace managers (e.g., namespacemanagers 2401 and 2402). Each of the namespace managers 2401, 2402,2403, 2411, and 2412 can maintain a database of namespace information,such as, for example, what namespace managers or providers areinterested in which namespace branches.

Namespace federation infrastructure 2400 includes providers 2421, 2422,2423, 2424, 2426, and 2427. Each of the providers can be interested inone or more namespace branches in the namespace federationinfrastructure. Providers exchange namespace information with acorresponding namespace manager. For example, provider 2422 exchangesnamespace information with namespace manager 2411. A correspondingnamespace manager then facilitates transferring the namespaceinformation to other namespace managers. For example, namespace manager2411 can transfer the namespace information to namespace manager 2401and namespace manager 2401 can in turn transfer relevant portions of thenamespace information to namespace managers 2402 and 2403.

A namespace federation infrastructure (e.g., namespace federationinfrastructure 2400) facilitates distributing lookup requests overnamespaces to appropriate providers. For example, it may be thatproviders 2801, 2802, and 2803 are each one of the providers 2421, 2422,2423, 2424, 2426, or 2427.

Namespace managers can federate using a variety of different mechanisms.A first federating mechanism includes peer namespace managers forwardingnamespace information to all other peer namespace managers. When anamespace manager is to join a namespace federation infrastructure, thenamespace manager utilizes a broadcast/multicast discovery protocol,such as, for example, WS-Discovery to announce its presence (abroadcast/multicast Hello) and issues a broadcast/multi-cast Probe todetect other namespace managers. The namespace manager then establishesa simple forwarding partnership with other namespace managers alreadypresent on the network and accepts new partnerships with newly joiningnamespace managers. Thereafter, the namespace manager can forward everynamespace request to its partners.

A second federating mechanism includes peer namespace managersefficiently forwarding all namespace information to other peer namespacemanagers. When a new namespace manager is to join a namespace federationinfrastructure, the new namespace manager utilizes a broadcast/multicastdiscovery protocol, such as, for example, WS-Discovery to announce itspresence (a broadcast/multicast Hello) and issues a broadcast/multicastProbe to detect other namespace managers that are part of the namespacefederation infrastructure. Upon detecting another namespace manager, thenew namespace manager establishes a partnership with the other namespacemanager. From the established partnership, the new namespace managerlearns about the presence of other namespace managers alreadyparticipating in federation namespace infrastructure. It thenestablishes partnerships with these newly-learned namespace managers andaccepts any new incoming partnership requests.

Both namespace manager arrivals/departures and namespace registrationsare flooded through the namespace federation infrastructure resulting inevery namespace manager having global knowledge of other namespacemangers and namespace registrations. With such global knowledge, anynamespace manager can forward lookup requests to only partners that haveproviders/subscribers registered under the namespace branch specified inthe request.

A third federating mechanism includes peer namespace managers indirectlyforwarding namespace information to other peer namespace managers. Inthis third mechanism, namespace managers are assigned unique identifiers(ID's), such as, for example, a 128-bit or 160-bit ID. The namespacemanager responsible for a given namespace tree is determined to be theone whose ID is closest to the one obtained by an at least one-waymapping function, such as, for example, hashing the given namespacetree. Such a hashing based mapping scheme for namespaces is described infurther detail below.

In this third mechanism, namespace manager arrivals and departures areflooded over the fabric. On the other hand, namespace registrations areforwarded to the namespace manager determined to be responsible for thenamespace branch specified in the request. For scalability, loadbalancing, and fault-tolerance, the namespace manager receivingnamespace registrations may reliably flood these registrations amongthose namespace mangers that are within its neighborhood set. Theneighborhood set for a specified namespace manager is determined to bethe set of namespace managers having IDs within a predefined range oneither side of the ID of specified namespace manager within a finitemodulo ID-address-space.

Similar to mechanism 2, a newly joining namespace manager utilizes abroadcast/multicast discovery protocol, such as, for example,WS-Discovery to announce its presence (a broadcast/multicast Hello) andissues a broadcast/multicast Probe to detect a namespace manager that isalready part of the namespace federation infrastructure. The newnamespace manager establishes a partnership with the discoverednamespace manager and uses that partnership to learn about the presenceof other namespace managers participating in the namespace federationinfrastructure. The new namespace manager then establishes furtherpartnerships with the newly discovered namespace managers and acceptsany new incoming partnership requests. It accepts incoming namespaceregistrations from its partners under the namespace branches for whichit is responsible and may flood them over its neighborhood set.

In response to incoming lookup requests, the new namespace managerconsults its registration database and forwards the requests to thenamespace managers having providers/subscribers registered under thenamespace branch specified in the request. Thus, when using this thirdmechanism, every namespace manager in the namespace federationinfrastructure has global knowledge of all other namespace managers butthe registration information is efficiently partitioned among thenamespace mangers. A namespace manager thus indirectly forwards thelookup request to only those partners that have providers/subscribersregistered under the namespace branch specified in the request. Thisindirection is accomplished via the namespace manager that has globalknowledge of the namespace registrations under the namespace branchspecified in the request.

A fourth federating mechanism includes peer namespace managersindirectly routing namespace information to other peer namespacemanagers. This fourth mechanism differs from the third mechanism in thesense that both namespace manager arrivals/departures and namespaceregistration/lookup requests are all routed instead of being flooded.Routing protocols are designed to guarantee rendezvous between namespacelookup requests and namespaces registration requests.

FIG. 25 illustrates an example of a computer architecture 2500 thatfacilitates routing requests indirectly to partners. Computerarchitecture 2500 depicts different types of computer systems anddevices potentially spread across multiple local discovery scopesparticipating in a namespace federation infrastructure.

Workstation 2533 can include a PnP provider instance that registers witha corresponding namespace manager under thelocation:/architecture200/scope221/Devices namespace branch. To informits partners of the presence of this PnP provider instance, workstation2533 routes namespace registration request 2501 over the namespacefederation infrastructure. Namespace registration request 2501 isinitially forwarded to laptop 2531, which in turn forwards namespaceregistration request 2501 to message broker 2537, which in turn forwardsnamespace registration request 2501 to message gateway 2541. Messagegateway 2541 saves the registration information registration request2501 in its database and returns success message 2504 to workstation2533.

Subsequently, another provider instance, this time that of runningservices, comes alive within the workstation 2533 and registers itselfwith the corresponding namespace manager under thelocation:/architecture200/scope221/Services namespace branch. This timethe namespace manager is aware that message gateway 2541 is responsiblefor registrations under location:/architecture200 and forwardsregistration request 2505 to message gateway 2541 directly. Messagegateway 2541 saves the registration information registration request2505 in its database and returns success message 2506 to workstation2533.

Subsequently, the printer 2536 (e.g., a UPnP printer) is powered on andsends announcement 2507. Server 2534 detects announcement 2507, assignsthe namespace location:/architecture200/scope224/Devices to printer2536, and routes registration request 2508 to message broker 2537.Message broker 2537 forwards registration request 2508 to messagegateway 2541. Message gateway 2541 saves the registration informationregistration request 2508 in its database and returns success message2591 to server 2534.

Subsequently, personal computer 2542 issues find request 2592 todiscover all devices under the Namespace branchlocation:/architecture200. Since personal computer 2542 doesn't knowwhere to forward find request 2592, it routes find request 2592 throughworkstation 2543. As the routing protocol essentially guaranteesrendezvous between registration and lookup requests for a givenNamespace tree, workstation 2543 forwards find request 2592 to messagegateway 2541. Message gateway 2541 forwards find request 2592 to boththe workstation 2533 and server 2534. Workstation 2533 and server 2534send response messages 2514 and 2516 respectively to personal computer2542.

This fourth mechanism works by routing a request to the namespacemanager (message gateway 2541) that has global knowledge of thenamespace registrations under the namespace branch (e.g.,location:/architecture200) specified in a request. This fourth mechanismessentially guarantees that routing can be accomplished in O(log N)hops, where N is the number of namespace managers participating in thefederation namespace infrastructure. Since this fourth mechanismefficiently partitions namespace registration information and does notmandate global knowledge of all the participating namespace managers, itscales to very large networks, even the Internet.

FIG. 26 illustrates an example of a binary relation between namespacemanages in a namespace federation infrastructure. The binary relationdepicted in FIG. 3 is one relation that may be utilized to implementmore efficient routing between namespace managers. Namespace managersparticipating in a namespace federation infrastructure are organized asa sorted list using a binary relation that is reflexive, anti-symmetric,transitive, total, and defined over the domain of namespace manageridentities. Both ends of the sorted list are joined, thereby formingring 306. This makes it possible for each namespace manager in thesorted list to view itself as being at the middle of the sorted list.The sorted list can be doubly linked so that any namespace manager cantraverse the sorted list in either direction. Further, there is a 1:1mapping from the value domain of the namespace manager identities (e.g.,2, 50, or 151) to the namespace managers themselves. This mappingaccounts for the sparseness of the namespace managers in the valuedomain when the mapping is not tight.

Each namespace manager on ring 2606 can include a routing table thatfacilitates routing namespace information (e.g., registration and lookuprequests) to other namespace managers. An example routing table for thenamespace manager having ID 64 is depicted in FIG. 26. The routing tableindicates that the successor to ID 64 is ID 76. The successor can be inthe immediate adjacent namespace manager in a clockwise direction fromID 64 on ring 2606. The successor can change, for example, when a newnamespace manager (e.g., with an ID of 71) joins or an existingnamespace manager (e.g., ID 76) leaves the namespace federationinfrastructure.

The routing table indicates that the predecessor to ID 64 is ID 50. Thepredecessor can be the immediate adjacent namespace manager in acounterclockwise direction from ID 64 on ring 306. The predecessor canchange, for example, when a new namespace manager (e.g., with an ID of59) joins or an existing namespace manager (e.g., ID 50) leaves thenamespace federation infrastructure.

The routing table indicates that the neighbors to ID 64 are IDs 83, 76,50 and 46. Neighbors can be identified using the larger of the twofactors size and range. A namespace manager is identified as a member ofa neighborhood when a corresponding ID is within minimum range of thesubject ID (e.g., in a clockwise or counterclockwise direction of ring2606) or there are less than some configured minimum neighborhood sizepresent in the neighborhood already. For example, on ring 306, thespecified range can have a magnitude of 20 and the size can be greaterthan 4. Accordingly, IDs within 20 locations of ID 64 in both theclockwise (+10) and counterclockwise direction (−10) are neighbors to ID64. The neighbors can change, for example, when namespace mangers joinor leave the namespace federation infrastructure or when the specifiedrange is changed. For example, with size equals 4 a new namespacemanager with ID 48 can replace the namespace manager having ID 46.

The routing table indicates that ID 64 can route directly to IDs 200, 2,30, 46, 50, 64, 76, 83, 98, and 135. Thus, when namespace manager havingID 64 receives a request, the namespace manager can route the requeststo the namespace manager having an ID in the routing table that iscloser to the namespace manager ID in the request.

FIG. 27 illustrates an example of an architecture 400 that facilitatesintegrating namespace federation infrastructure with other protocols. Anamespace federation infrastructure can support a provider-basedextension model. Accordingly, a federation namespace infrastructure canbe integrated with existing protocols provided the resource model of theexisting protocol is compatible with that of namespaces. Architecture2700 depicts namespace managers 2701, 2704, 2706 (e.g., of a namespacefederation infrastructure) interoperating with active directory 2702 andUDDI server 2703. The solid arrows indicate that namespace managerscommunicate using namespace federation protocols, the dashed areasindicate that namespace manages communicate with active directory 2702using LDAP protocols, and the dotted arrows indicate that namespacemanagers communicate with UDDI server 2703 using UDDI protocols.

Pub-sub topics is another example usage of namespaces. A pub-sub topiccan be viewed as a collection of subscribers to that topic; as such, thetopic name is treated as a namespace. An advantage of treating pub-subtopics as namespaces is that the namespace federation infrastructure canbe used to route notification messages from publishers to subscribers. Asubscription to a topic can be viewed as a namespace registrationrequest and publish to topic can be viewed as a namespace lookuprequest.

In some embodiments, a namespace federation infrastructure can provide abus-like abstraction to programmers for developing distributedapplications. For example, the namespace federation infrastructure canabstract liveness—the mechanism applications employ to know when aresource they are interested in has fallen off the network. To track agiven resource, the application subscribes to notifications sent to thepub-sub topic named after that resource's identity URI (i.e., its name).Any component (e.g., application) that notices that a given resource hasfallen off the network can publish a liveness notification message tothe topic named after the resource's identity URI, thereby informingother applications interested in tracking the resource. Since pub-subsubscriptions are federated across the namespace infrastructure andsince many identity schemes are hierarchical (to capture the containmentaspect of the resource from a liveness perspective), the system avoidsthe n² pinging problem of simple detection systems and scales very well.Further, the more interest components (e.g., applications) have in agiven resource, the quicker someone will notice that it has fallen offthe network, which is advantageous.

Developers can view a namespace federation infrastructure as a cloudinto which resources, such as, files and event sources are registered.Applications can issue find requests against the cloud to discoverregistered resources. Applications can also request the cloud tosubscribe on their behalf to both current and future event sourcesregistering with the cloud. Further, applications can subscribe topub-sub topics maintained in the cloud. Anyone can publish anotification message and the cloud takes care of forwarding the messageto the subscribers of the event topic into which that message waspublished.

Various types of resources can be published in Namespaces, includingservices, devices, files, hosts, components, items in a database,metadata about metadata (schemas), and so on. A resource can have aservice component hosting/backing it. For example, a file resource canhave a file server as the service component for accessing the file. Aconference room can have a receptionist's mailbox as the servicecomponent for scheduling a meeting.

Each resource can be associated with a resource descriptor that capturesits descriptive aspect. Thus, Resource descriptors can be queried toidentify resources of interest. When a resource is identified, theresource can be accessed through the resource's corresponding serviceaspect. The types of messages that can be sent to the servicehosting/backing a resource vary from one resource type to another. Forexample, file servers support opening file resources and receptionistsaccept scheduling requests for conference rooms.

The data model for implementing resource descriptors can be versionable,extensible, and interoperable. Such a resource data model can be sharedacross many of the current frameworks such as Distributed File System(“DFS”), AD, and UDDI. Such a single shared data model can facilitate ADobjects and DFS files (or resources form other resource managementsystems) being viewed as resources, federated using the namespacesapproach, and that are accessed by sending messages to the serviceshosting them.

Accordingly, resources can be defined to have the following properties:

-   -   Resource ID: a URI that can optionally be augmented with a set        of reference properties and can be stable in space and time. It        can be represented as an instance of a resource reference        schema. A resource ID along with resource properties can        collectively represent the identity of a resource.    -   Descriptor: a resource-specific schema instance containing        semi-static metadata about the resource. This metadata is useful        for resource selection. Resource descriptor schemas can be        taxonomized.    -   Config number: a monotonically-increasing number that identifies        a particular version of resource description data. This number        is incremented whenever the resource description is modified.    -   Instance ID: a monotonically-increasing number that identifies a        particular instance of an active resource. For example, this can        be the same as the boot time for service/device resources or the        file modification time for file resources.

With further reference to descriptors, a device can have metadata inaccordance with one or more schemas. For example, a printer can havemetadata in accordance with different schemas that describe differentaspects of the printer. Resource descriptor schemas can be standardizedby organizations such as UPnP Forum working groups (e.g., Printer Schemacan be standardized by the UPnP printer working group) and W3C. FIG. 30depicts example taxonomy 3000 for describing a resource. Within taxonomy3000 different schemas are generally represented as follows:

-   -   Service Reference Schema: extends a resource reference schema        and specifies a list of behavior types identifying messages        supported by the resource, a policy container for its assertions        (such as supported transports), and a set of extensions.    -   Resource Descriptor Schema: extends a resource reference schema        and specifies the descriptor's configuration number (see below        for explanation), friendly name of the resource, the service        reference of the service backing the resource, and a set of        extensions.    -   Namespace Node Descriptor Schema: extends a resource descriptor        schema and specifies the resources reachable from it as        instances of an edge descriptor schema.    -   Edge Descriptor Schema: specifies a locally-scoped edge name,        the edge type, and target resources.    -   Device Descriptor Schema: extends a resource descriptor schema        and specifies the serial number and manufacturer name.    -   Printer Descriptor Schema: extends the device descriptor schema        and specifies printer-specific properties such as resolution,        ability to print in color, pages per minute, and supported paper        sizes.

Any of the information defined in any of the above description schemascan be included in a query for identifying resources in a federationnamespace infrastructure. For example, the descriptor data can besearched and navigated using a filter (or query) expression. Forexample, one can filter by the type of descriptor schema or fieldvalues, navigate to instances reachable from its reference fields, applya sub-filter on those, and so on. In some embodiments, XPath-basedfilter expressions are used. Referring back to FIG. 29, using an XPathsyntax, a filter expression that operates over the description dataspecified by the resource description schemas can be used to locate aprinter in Location:/Bldg42/Floor1 that can print in color.

A namespace can specify a filter expression, in the form of a URIsegment parameter, for the fields/attributes defined on a namespace noderesource for selection and traversal. For example, the namespaceLocation:/Bldg42/Floor1/Room1226;employee=“employee1”/printer wouldtraverse namespace node resource “Room 1226” only if the descriptor of“Room 1226” has a “employee” field with the value “employee1”.Similarly, the namespaceOrganization:/Product/DevicesTeam;building=“Bldg33”/Dev/Computer2904;printer=“color”would traverse namespace node resource “Devices Team” only if itsdescriptor has a “building” field with the value “Bldg 33” (thusidentifying a first portion of resources) and would select namespacenode resource “Computer 2904” only if its descriptor has a “printer”field with the value “color” (meant to identify that a color printer hasbeen attached to it).

As previously described, namespace managers can be assigned a uniquenumeric identifier, such as, for example, a 160-bit ID. In someembodiments, the unique identifier is generated from hashing one or morenamespace manager characteristics, such as, for example, Domain NameServices (“DNS”) name, location, department, etc. Any of a variety ofdifferent hashing functions, such as, for example, SHA, can be used togenerate a unique ID.

Utilizing the unique namespace manager IDs the following functions canbe provided for routing namespace information in a namespace federationinfrastructure:

-   -   RouteNumerically(V, Msg): Given a value V from the value domain        of namespace manager identities and a message “Msg,” deliver the        message to namespace manager X whose identity can be mapped to V        using the mapping function.    -   Neighborhood(X, S): Neighborhood is the set of namespace        managers on the either side of namespace manager X (e.g., on        ring 306) with cardinality equal to S.

Embodiments of the present invention can also utilize proximitycriterion of namespaces managers participating in a federation.Proximity criteria can be defined as an equivalence relation thatpartition the set of federating namespace managers into a disjoint setof classes (or partitions). Generally, a relation R on a set S is anequivalence relation if it satisfies the following properties:

-   -   Reflexive: x in an element of S→x R x    -   Symmetric: Given x, y elements of S, x R y→y R x    -   Transitive: Given x, y, z elements of S, x R y        y R z→x R z

Embodiments of the present invention can support a plurality ofdifferent proximity criteria and proximity criteria can be arranged in apartial order. For example, a criterion that considers all the namespacenode resources belonging to “Corporation 1” to be proximally closeprecedes the criterion that considers all the namespace managers within“Corporation 1, Location A” to be proximally close. This results fromset of namespace managers considered proximally close by the formercriterion (belonging to “Corporation A”) being a super set of the set ofnamespace managers considered proximally close by the latter criterion(belonging to “Corporation 1, Location A”). On the other hand, there isno ordering relationship between the criterion that considers all thenamespace managers within “Corporation 1, Location A” as proximallyclose and the criterion that considers all the namespace managers within“Corporation 1, Location A” as proximally close.

Taking proximity considerations into account when computing routingnamespace managers for each namespace manager in a federation results inan increased chance that each routing hop on the path to the finaldestination remains within the proximity of the namespace manager thatoriginated the request. Further, significant progress in closing thedistance between the namespace managers in the numerical space can stillbe made.

Utilizing unique IDs along with proximity criterion, the followingadditional function can be provided for routing namespace information ina namespace federation infrastructure:

-   -   RouteProximally(V, Msg, P): Given a value V from the domain of        namespace manager identities and a message “Msg,” deliver the        message to the namespace manager Y whose identity can be mapped        to V among the namespace managers considered equivalent by the        proximity criteria P.

When a provider/subscriber registers at namespace branch with anamespace manager, the registration request is sent (and potentiallyrouted) to a partner namespace manager responsible for maintainingregistration information for the namespace tree specified in theregistration request. Alternately, it may be that the namespace managerthat originates the namespace registration request into the fabric isthe responsible namespace manger. Thus, embodiments of the inventioninclude routing a namespace registration request.

Routing a namespace registration request can include an act of receivinga namespace registration request to register a namespace branch, thenamespace registration request including a namespace identifier thatidentifies the namespace branch. For example, namespace manager 2412 canreceive registration request 2432, including a namespace ID 2442, fromprovider 2431. Since, namespace manager 2412 is not a peer namespacemanager, namespace manager 2412 can forward registration request 2432 tonamespace manager 2402. Namespace manager 2412 can canonicalizenamespace ID 2442 per rules identified by its scheme before transferringregistration request 2432 over the namespace federation infrastructure2400.

Routing a namespace registration request can include an act ofgenerating an at least one-way equivalent identification value based onthe scheme portion of the namespace identifier along with at least partof the path portion of the namespace identifier. For example, namespacemanager 2402 can generate a hash 2452 based on the scheme portion ofnamespace ID 2442 along with at least part of the path portion ofnamespace ID 2442. Any of a variety of different hashing functions, suchas, for example, SHA, can be used to generate a hash value from portionsof a namespace string. Generating a hash value for a namespace stringcan vary based on the configuration of the namespace federationinfrastructure.

For non-hierarchical Namespace schemes such as “uuid” (e.g., identifiedby the absence of the “:1” character sequence after the scheme), a hashcan be generated over the entire Namespace. For example, the entirenamespace string “uuid:a36fab9c-9c7f-42c3-97d8-36cd57e9bd29” may be usedto generate a SHA hash value.

Hierarchical namespaces can be authoritative or non-authoritative, withthe two distinguished, for example, by the respective charactersequences “://” and “:/” following the scheme component. Forauthoritative namespaces such as “name”, a hash is generated over thescheme part, followed by the “://” character sequence, the authoritycomponent, and the first path component of the namespace. For example,the portion “name://red.prn.xrx:200/printers” of the namespace string“name://red.prn.xrx:200/printers/b42-1749-a” may be used to generate aSHA hash value. For non-authoritative namespaces such as the “location”scheme of FIG. 6, a hash can be generated over the scheme part, followedby the “:/” character sequence and the first path component of theNamespace. For example, the portion “location:/Bldg42” of the namespacestring “location:/Bldg42/Floor1/Room1226”

Routing a namespace registration request can include an act of sendingthe namespace registration request to a namespace manager having anidentifier that is numerically closer to the at least one-way equivalentnumeric identification value than the identifiers of other namespacemanagers. For example, namespace manager 2402 can invoke aRouteNumerically function supplying hash 2452 and registration message2432 as input, for example, RouteNumerically(hash 2452, registrationmessage 2432). Alternately, a RouteProximally function can be used. Insome embodiments, the namespace registration request is sent directlyand no routing occurs.

Federation namespace infrastructure 2400 then utilizes federationprotocols to forward the registration message to the appropriatenamespace manager. For example, registration request 2432 can be routedto namespace manager 2403. It may be that namespace manager 2403 hasmigrated responsibility for the namespace branch to another namespacemanger. Thus, it may be that namespace manager 2403 returns a referralmessage to namespace manager 2402. Accordingly, when responsibility forthe namespace branch has been referred, namespace manger 2402 canreceive a referral message specifying the appropriate namespace manager.Namespace manager 2402 can in turn send registration request 2432 to theappropriate namespace manager. One or more referrals can be encountereduntil a namespace manager accepts or rejects the registration request.

Routing a namespace registration request can include an act ofassociating the namespace manager with the namespace branch. Forexample, namespace manager 2403 can be associated with the namespacebranch identified by namespace ID 2442 (through provider 2431).Namespace ID 2442 can, for example, identify a portion of namespace 2901or namespace 2902. Associations between a namespace manager and anamespace branch allow requests (e.g., lookup requests) specifying anamespace branch beneath the one specified in the registration requestto be forwarded, instead of being routed, to the namespace managerspecified in the association. Associations are broken when either anamespace manager failure is detected or a referral to a differentnamespace manager is obtained. When a failure is detected, subsequentrequests are routed until a new association can be formed.

Embodiments of the invention include migrating a namespace registrationrequest. Migrating a namespace registration request can include an actof determining that a namespace manager has satisfied a policyconstraint. For example, namespace manager 2403 can determine that theamount of namespace information (related to federation namespaceinfrastructure 2400) being processed at namespace manager 103 hasexceeded a configured threshold. A configured threshold can be, forexample, a total number of registrations maintained at a namespacemanager or a total number of lookup requests being serviced at anamespace manager.

Migrating a namespace registration request can include an act ofidentifying a namespace branch that can be migrated to meet a policyaction associated with the policy constraint. For example, namespacemanager 2403 can identify a namespace branch (e.g., corresponding tonamespace ID 2442) that can be migrated to reduce the namespaceinformation processed at namespace manager 2403 below a configuredthreshold. It may be that a namespace manager identifies a moreheavily-populated and/or heavily-serviced namespace branch formigration.

Migrating a namespace registration request can include an act of an actof migrating existing registrations for the namespace branch to apartner namespace manager in response to the policy action. For example,namespace manager 2403 can migrate existing registrations to a partner(e.g., a neighborhood) namespace manger in response to an action that isto occur to relieve the burden on the heavily-populated and/orheavily-serviced namespace branch.

Migrating a namespace registration request can also include an act ofreceiving a namespace request corresponding to the namespace branch. Forexample, namespace manager 2403 can receive registration request 2432correspond to the namespace branch represented by namespace ID 142.

Migrating a namespace registration request can also include an act oftaking action to redirect the namespace request to a partner namespacemanager. For example, as indicated by the dotted arrow, namespacemanager 2403 can reroute registration request 2432 to namespace manager2401. A namespace manager that migrates a namespace branch can invoke aRouteNumerically to reroute request to a different namespace manager.For example, RouteNumerically(H, migrateMsg) can be invoked to rerouterequests to a namespace manager (e.g., namespace manager identified byan at least one-way equivalent value of the namespace branch beingmigrated. For example, to migrate the branch location:/Bldg42/Floor1,namespace manager 2403 generates a hash H over the string“location:/Bldg42/Floor1”, invokes RouteNumerically(H, migrateMsg) toidentify the namespace manager 2401 responsible for the migrated branch,and migrates all the namespace registrations underneath the migratedbranch such as location:/Bldg42/Floor1/Room1226 andlocation:/Bldg42/Floor1/Room1119 to the identified namespace manager2401.

A namespace manager may also decide to forward all the namespaceregistrations encountered along the spine of the migrated namespacebranch to the partner namespace manager hosting the branch. Thisfacilitates the partner namespace manager branch servicing all thelookup requests that specify the namespace branch without the requestshaving to go through the migrating namespace manager, either directly orindirectly, all the time. The migrating namespace manager can leavebehind a stub indicating that it has migrated registration informationunder the specified namespace branch. The migrating namespace managercan also revoke subscriptions, if any, for liveness notificationstracking providers/subscribers specified in the migrated registrations.Accordingly, subsequent namespace registrations under and along thespine of the migrated namespace branch received by the migratingnamespace manager are forwarded to the partner namespace manager.

Embodiments of the invention include processing a namespace registrationrequest. Processing a namespace registration request can include an actof receiving a namespace registration request to register a namespacebranch, the namespace registration request including a namespace URIstring that identifies the namespace branch and a unique reference oridentifier for the provider (or subscriber) requesting registration inthe namespace branch. For example, namespace manager 2403 can receiveregistration request 2432 that includes reference to provider 2431.

Processing a namespace registration request can include an act ofdetermining that a namespace manager is interested in the namespacebranch. For example, namespace manager 2402 can determine if namespacemanager 2402 is responsible for the namespace branch represented bynamespace ID 142 (e.g., Organiztion:/Product/Messaging Team). Whennamespace manger 2402 is not responsible, namespace manager 2402 canforward the namespace registration request (e.g., registration request2432) to a responsible namespace manger (e.g., namespace manger 2403)for the specified namespace branch. Alternately, when namespace manger2402 is not responsible, namespace manager 2402 can send a referralmessage 2434 to the namespace manger (e.g., namespace manager 2403) thatinitiated the registration request (e.g., registration request 2432) toinstead contact the responsible namespace manager (e.g., namespacemanager 2401). When namespace manager 2402 is responsible, namespacemanager 2402 can retain the namespace registration request.

Processing a namespace registration request can include includes an actof saving the namespace identifier in an appropriately indexed namespaceregistration database. For example, if the namespace identifier is a URIstring, it is stored in the namespace registration database index inalphabetical order with longer strings ranked higher. For example,namespace manager 2403 can save namespace ID 142 in namespaceregistration database. The dashed line and corresponding dashed boxsurrounding provider 2431, indicates that namespace manager 2403 hasreferenced provider 2431 as being interested in the namespacerepresented by namespace ID 142.

Processing a namespace registration request can also include an act ofdetermining how often the liveliness of the provider is to besubsequently verified. For example, namespace manager 2403 can determinehow often the liveliness of the provider 2431 is to be subsequentlyverified. Namespace provider 2403 can optionally subscribe to livenessnotifications published to the pub-sub topic of provider 2431 identifiedby ID 161. The pub-sub topic can be identified by ID 161. Alternately,if a liveness subscription is not made, the registration is assigned atime-limited lease. Provider 2431 can renew registration before thelease expires by directly contacting namespace manager 2403. Otherliveness mechanisms can also be used.

Namespace manager and provider liveliness can be distributed across ahierarchy. A namespace manager positioned at a higher level in ahierarchy can rely on other similarly positioned namespace managers toreport liveness information for corresponding lower level namespacemanagers and providers. For example in FIG. 24, namespace manager 103can track the liveness of namespace manager 2402 (both are rootnamespace managers). Namespace manager 2403 can rely on namespacemanager 2402 to report failures of any corresponding lower levelnamespace managers (e.g., namespace manager 2412) or providers (e.g.,provider 2424). Namespace manager 2402 would in turn rely on namespacemanager 2403 to report similar type failures (e.g., failure of provider2426).

Subsequent to a successful registration (or failure) of provider 2431,namespace manager 2402 can send a message indicating the success (orfailure) to provider 2431.

From time to time, consumers (other computer systems or devices) maydesire access to resources in a namespace branch that is managed by aprovider. To obtain access to resources, the consumers may issue lookuprequests to attempt to identify resources. Lookup requests can bereceived at namespace mangers and delivered to one or more appropriateproviders. Generally, when a namespace manager receives a lookuprequest, it routes that lookup request to the partner namespace managerclosest to it (as determined by some predefined proximity metric) andtoward the neighborhood of the namespace manager responsible for thenamespace branch specified in the request. As the registrationinformation is replicated across the neighborhood namespace managers,the lookup request can be satisfied by any namespace manager in theneighborhood set.

Routing via the namespace manager closest to the namespace mangeroriginating the lookup request results in improved network throughputand dynamic load balancing, since lookup requests get automatically andefficiently partitioned across the neighborhood namespace managers fromthe lookup request satisfaction perspective. To facilitate routing, thealgorithm for mapping namespace IDs specified in lookup requests can beessentially the same as the algorithm for mapping namespace IDsspecified in registration requests. For example, a 1:1 mapping from thevalue domain of the namespace identities to the namespace managers canbe used to map namespace IDs for both lookup and registration requests.

Embodiments of the invention include routing, migrating, and processingnamespace lookup requests. Methods similar to those used for namespaceregistration requests can also be used for namespace lookup requests.

Embodiments of the invention include a resource participating inmultiple namespaces. A resource participating in multiple namespaces caninclude an act of establishing a unique resource identifier for aresource, including establishing a path portion of a URI thatcorresponds to the resource. For example, an identifier of “printer2903” can be established for a printer.

A resource participating in multiple namespaces can include an actpublishing the availability of the resource in a first namespace. Forexample, printer 2903 can publish its availability in namespace tree2901. A resource participating in multiple namespaces can include an actof linking the unique resource identifier to a first namespace noderesource in the first namespace such that the first namespace can betraversed to identify the resource. For example, namespace segment 2931can be established to link printer 2903 to the “Dev Team” namespace noderesource. Accordingly, namespace tree 2901 (and the “Dev Team” namespacenode resource) can be traversed to identify printer 2903.

A resource participating in multiple namespaces can include an actpublishing the availability of the resource in a second namespace. Forexample, printer 2903 can publish its availability in namespace tree2902. A resource participating in multiple namespaces can include an actof linking the unique resource identifier to a second namespace noderesource in the second namespace such that the second namespace can betraversed to identify the resource. For example, namespace segment 2932can be established to link printer 2903 to the “Room 1226” namespacenode resource. Accordingly, namespace tree 2902 (and the “Room 1226”namespace node resource) can also be traversed to identify printer 2903.

Embodiments of the invention include identifying a subset of resourcesin the namespace federation infrastructure. Identifying a subset ofresources in the namespace federation infrastructure can include an actreceiving a query from a device. For example, a provider for namespacetree 2902 can receive a query from a device that is network connectableto the provider. The query includes a first query portion identifying afirst portion of resources that satisfies first query criteria at afirst level in a namespace hierarchy. For example, a first query portioncan identify a first portion of resources that satisfies first querycriteria after traversing namespace segment “Floor 2” (in namespace tree2902). A first portion of resources can be, for example, employees, andfirst criteria can also include, for example, assigned to the “MessagingTeam”. Thus, the first query portion can identify all the employeesassigned to the “Messaging Team” that work on Floor 2 (of Bldg 42). Insome embodiments, the first query criteria are utilized to navigatethrough the properties of resources that reference the first portion ofresources.

The query includes a second query portion identifying a second portionof resources selected from among the resources included in the firstportion of resources. For example, a second query portion can identify asecond portion of resources that satisfies second query criteria aftertraversing namespace segment “Room 2005” (in namespace tree 2902). Asecond portion of resources can be, for example, administrators, andsecond criteria can be, for example, devices. Thus, the second queryportion can identify printer administrators with office cubicles in Room2005. In some embodiments, the second query criteria are utilized the tonavigate through the properties of the first portion of resources thatreference the second portion of resources.

Accordingly, providing the resources identified from the first queryportion as input to the second query portion, the results of thereceived query can (depending on the field definitions in the resourcesschemas) identify printer administrators with offices in 2^(nd) floor,Room 2005, and assigned to Messaging Team.

Identifying a subset of resources in the namespace federationinfrastructure can include an act of returning the identity of thesecond portion of resources to the device. For example, the provider fornamespace tree 2902 can return the identity of administrators of devicesin ConfRoom 2005 that are owned by Messaging Team employees on Floor 2to the network connectable device.

Embodiments of the invention include organizing a plurality ofresources. Organizing a plurality of resources can include an act ofdetermining that a new resource is to be included in one or morenamespaces, each of the one or more namespaces being configured toorganize one or more resources. For example, it can be determined thatprinter 2903 is to be included in namespace 2901 and/or namespace 2902.Organizing a plurality of resources can include an act of identifying afirst resource within a first namespace of the one more namespaces thatis to be related to the new resource. For example, it can be identifiedthat room 1226 in namespace 2902 is to be related to printer 2903.Similarly, it can be identified that Dev Team in namespace 2901 is to berelated to printer 2903.

Organizing a plurality of resources can include an act of using a firstnamespace segment to link the new resource to the first resource suchthat the namespace segment can be traversed to navigate from theexisting resource to the new resource within the namespace. For example,namespace segment 2932 can be used to link printer 2903 to Room 1226such that namespace segment 2932 can be traversed to navigate form Room1226 to printer 2903. Similarly, namespace segment 2931 can be used tolink printer 2903 to Dev Team such that namespace segment 2931 can betraversed to navigate form Dev Team to printer 2903.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or essential characteristics. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. The scope of the invention is, therefore, indicatedby the appended claims rather than by the foregoing description. Allchanges which come within the meaning and range of equivalency of theclaims are to be embraced within their scope.

What is claimed is:
 1. At a computer system in a federationinfrastructure comprising a ring of federated nodes, the computer systemincluding one or more processors and system memory, a method forproviding optimized access to a federation infrastructure resource, themethod comprising: an act of receiving a user request to access afederation infrastructure resource that is hosted at a first node in thering of federated nodes, the user request having been sent from arequesting user component; an act of detecting that the user request isnot directed to an optimized location within the ring of federated nodesfor accessing the federation infrastructure resource at the first node,the detecting being based on analyzing processing information forproviding optimized access to federation infrastructure resources andone or more of: one or more characteristics of the requesting usercomponent and one or more characteristics of the user request, theprocessing information including redirection information and priorityinformation for optimizing requests for the federation infrastructureresource; an act of determining an appropriately optimized locationwithin the ring of federated nodes for the requesting user component toaccess the federation infrastructure resource, including: when (i) theredirection information identifies a second node in the ring offederated nodes not hosting the federation infrastructure resource forlower-priority requests and (ii) the priority information identifies therequest as a lower-priority request, determining that the appropriatelyoptimized location is the second node not hosting the federationinfrastructure resource, and when (i) the redirection informationidentifies the first node for higher-priority requests and (ii) thepriority information identifies the request as a higher-priorityrequest, determining that the appropriately optimized location is thefirst node at which the federation infrastructure resource is hosted;and an act of sending an indication of the appropriately optimizedlocation within the ring of federated nodes for the requesting usercomponent to access the federation infrastructure resource based onanalyzing the processing information.
 2. The method as recited in claim1, wherein the act of receiving a user request to access a federationinfrastructure resource comprises an act of a node in the ring offederated nodes receiving the user request to access a federationinfrastructure resource directly from the requesting user component. 3.The method as recited in claim 1, wherein the act of sending anindication of an appropriately optimized location within the ring offederated nodes for the requesting user component to access thefederation infrastructure resource comprise an act of the node sendingan indication of the appropriately optimized location directly to therequesting user component.
 4. The method as recited in claim 1, whereinthe act of receiving a user request to access a federationinfrastructure resource comprises an act of a node in the ring offederated nodes receiving the user request to access a federationinfrastructure resource from a communication intermediary, thecommunication intermediary sending the user request on behalf of therequesting user component.
 5. The method as recited in claim 4, whereinthe act of detecting that the user request is not directed to anoptimized location within the ring of federated nodes for accessing thefederation infrastructure resource comprises an act of detecting thatthe node is not the optimized location for accessing the federationinfrastructure resource.
 6. The method as recited in claim 5, whereinthe act of sending an indication of an appropriately optimized locationwithin the ring of federated nodes for the requesting user component toaccess the federation infrastructure resource comprise an act of thenode sending an indication of the appropriately optimized locationdirectly to the communication intermediary.
 7. The method as recited inclaim 1, wherein the act of receiving a user request to access afederation infrastructure resource comprises an act of a communicationintermediary receiving a user request to access a federationinfrastructure resource, the user request directed to a node in the ringof federated nodes.
 8. The method as recited in claim 7, wherein the actof detecting that the user request is not directed to an optimizedlocation within the ring of federated nodes for accessing the federationinfrastructure resource comprises an act of the communicationintermediary detecting that the node is not the optimized location foraccessing the federation infrastructure resource.
 9. The method asrecited in claim 8, wherein the act of sending an indication of anappropriately optimized location within the ring of federated nodes forthe requesting user component to access the federation infrastructureresource comprise an act of the communication intermediary sending anindication of the appropriately optimized location to the usercomponent.
 10. The method as recited in claim 1, wherein the act ofsending an indication of an appropriately optimized location within thering of federated nodes for the requesting user component to access thefederation infrastructure resource comprises an act of sending adestination address for the appropriately optimized location to therequesting user component.
 11. At a computer system in a federationinfrastructure comprising a ring of federated nodes, the computer systemincluding one or more processors and system memory, a method foroptimizing access to a federation infrastructure resource, the methodcomprising: an act of detecting a component request to optimize accessto one or more federation infrastructure resources associated with thering of federated nodes so as to reduce inter-node communication costsbetween nodes on the ring of federated nodes when providing access tothe one or more federation infrastructure resources; an act ofdetermining that a hosting location of an associated federationinfrastructure resource, selected from among the one or more federationinfrastructure resources, is not optimized on the ring of federatednodes for providing one or more users that are associated with acomponent that sent the component request access to the associatedfederation infrastructure resource; in response to the componentrequest, an act of indicating that the hosting location of theassociated federation infrastructure resource on the ring of federatednodes is to be optimized to reduce inter-node communication costsbetween nodes on the ring of federated nodes when providing access tothe associated federation infrastructure resource; and in response tothe component request, an act of updating processing information forproviding optimized access to the associated federation infrastructureresource, the processing information including redirection informationand priority information for optimizing requests for the associatedfederation infrastructure resource, the redirection informationidentifying a node in the ring of federated nodes hosting the associatedfederation infrastructure resource for higher-priority requests and oneor more nodes in the ring of federated nodes not hosting the associatedfederation infrastructure resource for lower-priority requests, thepriority information usable for identifying priority level of requests.12. The method as recited in claim 11, wherein the act of detecting acomponent request to optimize access to one or more federationinfrastructure resources comprises an act of receiving a request tooptimize access to a federation infrastructure resource from acommunication service layer component.
 13. The method as recited inclaim 12, wherein the act of receiving a request to optimize access to afederation infrastructure resource from a communication service layercomponent comprises an act of receiving a request to optimize access toa federation infrastructure resource for one or more users having acommon interest in the federation infrastructure resource.
 14. Themethod as recited in claim 11, wherein the act of detecting a componentrequest to optimize access to one or more federation infrastructureresources comprises an act of detecting that an application system hasan application context associated with a resource hosted on the ring offederated nodes.
 15. The method as recited in claim 14, wherein the actof indicating that the hosting location of the associated federationinfrastructure resource on the ring of federated nodes is to beoptimized to reduce inter-node communication costs comprise an act ofsourcing an event to the application system.
 16. The method as recitedin claim 11, wherein the act of determining that the hosting location ofan associated federation infrastructure resource is not optimizedcomprises: an act of determining that the federation infrastructureresource is currently hosted at a first node on the ring of federatednodes; and an act of determining that inter-node communication costbetween nodes on the ring of federated nodes could be reduced if hostingof the federation infrastructure resource was moved to second differentnode on the ring of federated nodes.
 17. The method as recited in claim11, wherein the act of indicating that the hosting location of theassociated federation infrastructure resource on the ring of federatednodes is to be optimized comprises an act of indicating that the hostinglocation is to be moved from a first node on the ring of federated nodesto a second node on the ring of federated nodes.
 18. The method asrecited in claim 11, further comprising an act of optimizing the hostinglocation of the associated federation infrastructure resource on thering of federated nodes in accordance with the indication that thehosting location is to be optimized.
 19. A system, the systemcomprising: one or more processors; system memory; and one or morecomputer storage media having stored there one computer-executableinstructions for participating in a federation infrastructure, thefederation infrastructure including a ring of federated nodes and one ormore higher service layers, wherein the federation infrastructure isconfigured to provide optimized access to a federation infrastructureresource, by being configured to: receive a user request to access afederation infrastructure resource that is hosted at a first node in thering of federated nodes, the user request having been sent from arequesting user component; detect that the user request is not directedto an optimized location within the ring of federated nodes foraccessing the federation infrastructure resource, the detecting beingbased on analyzing processing information and one or more of: one ormore characteristics of the requesting user component and one or morecharacteristics of the user request, the processing informationincluding redirection information and priority information foroptimizing requests for the federation infrastructure resource;determine an appropriately optimized location within the ring offederated nodes for the requesting user component to access thefederation infrastructure resource, including: when (i) the redirectioninformation identifies a second node in the ring of federated nodes nothosting the federation infrastructure resource for lower-priorityrequests and (ii) the priority information identifies the request as alower-priority request, determining that the appropriately optimizedlocation is the second node not hosting the federation infrastructureresource, and when (i) the redirection information identifies the firstnode for higher-priority requests and (ii) the priority informationidentifies the request as a higher-priority request, determining thatthe appropriately optimized location is the first node at which thefederation infrastructure resource is hosted; and send an indication ofthe appropriately optimized location within the ring of federated nodesfor the requesting user component to access the federationinfrastructure resource based on analyzing the processing information;and wherein the federation infrastructure is configured to optimizeaccess to a federation infrastructure resource, by being configured to:detect a component request to optimize access to one or more federationinfrastructure resources associated with the ring of federated nodes soas to reduce inter-node communication costs between nodes on the ring ofnodes when providing access to the one or more federation infrastructureresources; determine that at least one of: a hosting location of or anaccess location for an associated federation infrastructure resource,selected from among the one or more federation infrastructure resources,is not optimized on the ring of federated nodes for providing one ormore users that are associated with a component that sent the componentrequest access to the associated federation infrastructure resource; andin response to the component request, indicate that the at least one ofthe hosting location of or the access location for the associatedfederation infrastructure resource on the ring of federated nodes is tobe optimized to reduce inter-node communication costs between nodes onthe ring federated nodes in when providing access to the associatedfederation infrastructure resource.
 20. A computer system comprising afederated node in a federation infrastructure, the federated nodeincluding: one or more processors; and one or more computer-readablemedia storing computer-executable instructions that, when executed bythe one or more processors, cause the federated node to provideoptimized access to a federation infrastructure resource, including thefollowing: receiving a user request to access a federationinfrastructure resource that is hosted at a first node in the ring offederated nodes, the user request having been sent from a requestinguser component; detecting that the user request is not directed to anoptimized location within the ring of federated nodes for accessing thefederation infrastructure resource at the first node, the detectingbeing based on analyzing processing information for providing optimizedaccess to federation infrastructure resources and one or more of: one ormore characteristics of the requesting user component and one or morecharacteristics of the user request, the processing informationincluding redirection information and priority information foroptimizing requests for the federation infrastructure resource;determining an appropriately optimized location within the ring offederated nodes for the requesting user component to access thefederation infrastructure resource, including: when (i) the redirectioninformation identifies a second node in the ring of federated nodes nothosting the federation infrastructure resource for lower-priorityrequests and (ii) the priority information identifies the request as alower-priority request, determining that the appropriately optimizedlocation is the second node not hosting the federation infrastructureresource, and when (i) the redirection information identifies the firstnode for higher-priority requests and (ii) the priority informationidentifies the request as a higher-priority request, determining thatthe appropriately optimized location is the first node at which thefederation infrastructure resource is hosted; and sending an indicationof the appropriately optimized location within the ring of federatednodes for the requesting user component to access the federationinfrastructure resource based on analyzing the processing information.21. A computer system comprising a federated node in a federationinfrastructure, the federated node including: one or more processors;and one or more computer-readable media storing computer-executableinstructions that, when executed by the one or more processors, causethe federated node to optimize access to a federation infrastructureresource, including the following: detecting a component request tooptimize access to one or more federation infrastructure resourcesassociated with the ring of federated nodes so as to reduce inter-nodecommunication costs between nodes on the ring of federated nodes whenproviding access to the one or more federation infrastructure resources;determining that a hosting location of an associated federationinfrastructure resource, selected from among the one or more federationinfrastructure resources, is not optimized on the ring of federatednodes for providing one or more users that are associated with acomponent that sent the component request access to the associatedfederation infrastructure resource; in response to the componentrequest, indicating that the hosting location of the associatedfederation infrastructure resource on the ring of federated nodes is tobe optimized to reduce inter-node communication costs between nodes onthe ring of federated nodes when providing access to the associatedfederation infrastructure resource; and in response to the componentrequest, updating processing information for providing optimized accessto the associated federation infrastructure resource, the processinginformation including redirection information and priority informationfor optimizing requests for the associated federation infrastructureresource, the redirection information identifying a node in the ring offederated nodes hosting the associated federation infrastructureresource for higher-priority requests and one or more nodes in the ringof federated nodes not hosting the associated federation infrastructureresource for lower-priority requests, the priority information usablefor identifying priority level of requests.